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About This Manual 


Objectives of This Manual 

This manual describes in detail the design, features, and correct use of the 
Network Interface (NI) chip of the Connection Machine CM-5 system. This 
description is at a level sufficient for low-level CM-5 coders to make full use of 
the NTs features. Both user- and supervisor-level information is included, as well 
as numerous programming examples written in the C programming language. 


intended Audience 

This manual is intended for use by knowledgeable CM-5 programmers. While 
it contains some overview information, this document is a reference manual, not 
a tutorial. This manual should be used in conjunction with other progra mmin g 
guides and with assistance from Thinking Machines Corporation representatives. 


Revision Information 

This manual is a consolidation of two previously published manuals: 
Programming the NI and NI Systems Programming. 

This manual replaces both of the earlier books. This manual contains the same 
information and examples, and reflects the most recent revision of the NI chip, 
used in the CM-5E version of the Connection Machine system node hardware. 
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and a description of the important features of the chip. 

A Generic Network Interface 
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work interfaces. 

The Data Network 

The registers and features of the three Data Network interfaces. 
The Control Network 

The registers and features of the three Control Network inter¬ 
faces (broadcast, combine, and global). 

NI Interrupts 

A description of the various interrupt classes of the NI, and of 
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NI Programming Issues 
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NI Interrupts 
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A list of all low-level programming constants defined by the 
files cmsys/ni_constants .h and cmsys/ni_defines .h, 
with the symbols grouped by register and field. 

Appendix E CMOS_signal Man Page 

The UNIX manual page for the CMOS_signal system call. 

Appendix F NI Accessor Examples 

A set of simple C code examples of routines that read and write 
NI registers and perform other useful functions. 


Appendix G Sample NI Programs 

C code examples demonstrating the NI features described in the 
chapters of this manual. 
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header files that define the CM Network Accessor interface. 


Appendix I NI Chip Version 2.2 Changes 

A quick-reference list of the changes to the NI chip as of Version 
2.2, with references into the main text of this manual. 

NI Memory Map 

A two-sided memory map of the NI registers and fields. 


Related Documents 

These documents are part of the Connection Machine documentation set: 

■ Connection Machine CM-5 Technical Summary, November 1993 

■ VU Programmer’s Handbook, CMOST Version 7.2 August 1993 
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Notation Conventions 

The table below displays the notation conventions observed in this manual. 


Convention 

Meaning 

bold typewriter 

UNIX and CM System Software commands, com¬ 
mand options, and filenames, when they appear 
embedded in text. Also, syntax statements and pro¬ 
gramming language elements, such as keywords, 
operators, and function names, when they appear 
embedded in text. 

italics 

Argument names and placeholders in function and 
command formats. 

typewriter 

Code examples and code fragments. 

% bold typewriter 

regular typewriter 

In interactive examples, user input is shown in 
bold typewriter and system output is shown in 

regular typewriter font. 


XX 
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Customer Support 


Thinking Machines Customer Support encourages customers to report er¬ 
rors in Connection Machine operation and to suggest improvements in our 
products. 

When reporting an error, please provide as much information as possible to 
help us identify and correct the problem. A code example that failed to exe¬ 
cute, a session transcript, the record of a backtrace, or other such 
information can greatly reduce the time it takes Thinking Machines to re¬ 
spond to the report. 

If your site has an applications engineer or a local site coordinator, please 
contact that person directly for support. Otherwise, please contact Thinking 
Machines’ home office customer support staff: 


Internet 

Electronic Mail: customer-suppor t@think. com 


UUCP 

Electronic Mail: ames! think! customer-support 


U.S. Mail: T hinkin g Machines Corporation 

Customer Support 
245 First Street 

Cambridge, Massachusetts 02142-1264 
Telephone: (617) 234-4000 
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Chapter 1 


The CM-5 Network Interface 



First, a word to the wise. You’re reading this manual for one of two reasons: 

■ You absolutely, positively must write programs that manipulate the net¬ 
work hardware of the CM-5 at the lowest possible level. 

■ You’ve heard about a CM-5 component called the “Network Interface,” 
and think it would be interesting to write a program that manipulates it. 

If it’s the latter, we strongly suggest that you consider using a higher-level pro¬ 
gramming method instead. Writing code at the level described in this manual 
means taking direct control of the Network Interface chip, the part of the CM-5 
hardware that manages the machine’s internal communications networks. This 
isn’t something that you should be doing unless you have no alternative. 

Also, be aware that code that directly accesses the Network Interface chip will 
not be supported in future software and hardware releases — your code may re¬ 
quire extensive modification to run. For essential code you should use the CMMD 
software interface instead. CMMD gives you nearly the same level of access to 
the CM-5 hardware, but provides it through a standard software interface that will 
be easily portable to future releases. (For more information, see the CMMD 
User’s Guide.) 

With this warning out of the way, we’ll assume that you’re reading this man ual 
for the first reason given above, and show you how to make use of the Network 
Interface (NI) chip. This manual presents the software tools that you need to pro¬ 
gram the NI and provides code examples throughout that show you how to do 
simple network operations on the CM-5. 
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1.1 The CM-5 System: Nodes and Networks 

The Network Interface chip, or NI, manages the internal communications net¬ 
works of the CM-5. Because the main focus of this manual is the Network 
Interface, it makes sense to start with an overview of the NTs location and func¬ 
tion within the CM-5 system. 


r 

Networks "i 




^Nodes '" 9 p p F p p p p p 
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'_ i 


Figure 1. The CM-5 system: Processing nodes, plus Data and Control Networks. 


The CM-5 contains a large number of processing nodes, which perform the arith¬ 
metic computations involved in a CM-5 program. The processing nodes are 
linked together by two internal communications networks, the Data Network and 
the Control Network. (See Figure 1.) The two CM-5 networks are similar in de¬ 
sign; both are scalable, high-speed data communications networks. However, the 
two networks have distinct intents and purposes. The Data Network is used for 
high-volume exchange of data between nodes. The Control Network is used to 
control and synchronize the operations of the nodes. 


1.1.1 The CM-5 Networks 
The Data Network 

The Data Network is a high-speed, high-bandwidth network designed to hanHfe 
the simultaneous node-to-node transmission of thousands of messages. The Data 
Network is composed of two halves, the left interface and the right interface , 
both of which are connected to all processing nodes. The left and right interfaces 
can be used either independently or together as the combined Data Network. 

Terminology Note: This combination of the left and right halves of the Data 
Network is sometimes called the “middle” interface by NI programmers. 
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The Control Network 

The Control Network is used for control tasks that require the joint cooperation 
of all nodes. It provides three separate functions: 

■ The broadcast interface distributes a single numeric value to every node. 
It consists of two subinterfaces: a user broadcast interface and a supervi¬ 
sor broadcast interface. 

■ The combine interface receives a single value from each node, combines 
the values arithmetically or logically, and then distributes the combined 
result to all nodes. 

■ The global interface handles global synchronization of the nodes. It con¬ 
sists of a number of distinct interfaces for synchronous and asynchronous 
messaging by user and supervisor (OS) code. 


For the Curious: The Diagnostic Network 

There is also a third major CM-5 network, the Diagnostic Network, used by the 
system manager to configure the CM-5 hardware and to diagnose hardware prob¬ 
lems. However, because the NI chip is not used to access it, the Diagnostic 
Network is not discussed further in this manual. 


1.1.2 Processing Nodes 

Each processing node contains a RISC microprocessor, a memory subsystem, 
and a Network Interface (NI) chip, linked together in a bus arrangement: 



Figure 2. The components of a typical processing node. 
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For the Curious: In the current implementation, the microprocessor is a SPARC 
chip; it executes both user and operating system (OS) code. The memory subsys¬ 
tem consists of DRAM memory controlled either by a single memory controller 
or by a set of four vector units (if your CM-5 has the vector unit option installed). 


1.1.3 Partitions and Partition Managers 

The processing nodes are grouped by software into partitions, with each partition 
monitored by a partition manager (PM). (See Figure 3.) Each partition can be as 
small as 32 nodes, or as large as the entire machine. The partitioning is controlled 
by the system administrator, who can create and alter partitions as needed. 
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Figure 3. A partition of nodes and its partition manager. 


The partition manager (PM) contains a RISC CPU and connecting hardware that 
allows the PM to interact with other computers and with users on terminals. 
Thus, the PM is the “gateway” by which a programmer gains access to the pro¬ 
cessing nodes of the CM-5 and instructs the CM-5 to execute a program. 

The PM is attached to the Data and Control Networks, and can communicate 
with its partition of processing nodes by sending and receiving messages via its 
own NI chip. Programs written for the CM-5 normally include two separate files 
of code, one for the PM and one for the nodes. 
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1.1.4 Programming Models 
User Programming Model 

From a user’s point of view, the CM-5 is the single partition of nodes associated 
with the PM that compiles and executes the user’s code. CM-5 programs often 
compile into two separate sets of code, one for the PM and one for the nodes. 

The PM and the nodes typically operate in a data parallel style: the nodes execute 
identical programs simultaneously, and the PM controls which function the nodes 
will execute next. (For more information on program structure, see Chapter 7.) 

The PM typically controls program flow, and handles all external interactions 
(communicating with the user by keyboard input and screen output, exchanging 
files and data with other computing systems over external networks, etc.). 

The nodes typically operate in an event-driven loop, waiting for instructions 
from the PM about which section of code to execute next. 


OS Programming Model 

From an OS point of view, the CM-5 is a set of partitions, each of which has a 
number of associated processes that can be swapped in. 

The CM-5 OS manages the execution and swapping of processes within parti¬ 
tions, as well as any exchange of data that takes place between partitions (for 
example, when a user program needs to read or write data from an I/O device). 

Under the CMOST operating system shipped with the CM-5, each PM runs a full 
and complete UNIX-based operating system, while each of the nodes runs a 
small kernel of OS code that is optimized for computation and communication. 
It is this kernel of code that provides the event-driven dispatch loop described in 
the user programming model above. 
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1.2 The NI Chip 

The NI chip is located between the RISC microprocessor and the two CM-5 net¬ 
works. Each network provides a specific set of network interfaces, and the role 
of the Network Interface chip is to make those interfaces available to the node 
microprocessor, and thereby to user and OS programs. 



Figure 4. NI provides access to Data Network and Control Network. 


When the microprocessor directs the NI to send a message via one of the net¬ 
works, the NI handles the dispatching of the message, and collects any replies 
from the networks. The NI uses send FIFOs (queues) to hold outgoing messages 
until they can be sent, and receive FIFOs to hold incoming messages until the 
microprocessor can read them. 


1.3 The NI Registers 

The NI chip is register-based. Its network functions are controlled entirely by 
reading and writing NI registers. Access to these registers is provided by 
memory-mapping — the NI registers are mapped into the microprocessor’s 
memory address space. Thus, from a programmer’s point of view the NI appears 
as a region of processor memory with some unique properties. 

The microprocessor can either directly use the registers of the NI chip to send and 
receive messages, or it can use indirect methods, such as having the NI signal an 
interrupt whenever a message arrives. (Interrupts can also be “broadcast” from 
one NI chip to all other NIs in a partition.) Control of the NI is therefore based 
on register operations, interrupts, and (in extreme cases) NI Resets, which are 
described later in this chapter. 
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The NI registers occupy a virtual memory region 512 Kbytes long. However, the 
NI registers are mapped into microprocessor memory twice, as two separate 
virtual memory areas: the user area and the supervisor area. (See Figure 5.) 


Processor Memory 




' ' ‘t, N .' '4/ 


Supervisor Area 

^ 

User Area 




Offset (in hex): 

0x100000 

0x080000 

0 (base address) 


Figure 5. The NI registers are mapped into user and supervisor memory areas. 


The user area occupies 512 Kbytes of virtual memory, startin g at the base address 
of the NI memory region (see Section 1.3.1). The supervisor area occupies the 
512 Kbytes immediately following the user area. 

The user and supervisor areas contain the same registers at the same offsets, but 
the hardware mapping is designed so that the NI registers for supervisor features 
are accessible only from the supervisor area. Any attempt to access supervisor 
registers from the user area signals a Bus Error. (A programmer sees this as a 
segmentation violation.) Thus, when this manual speaks of “the supervisor” per¬ 
forming an operation, or of an NI feature that is “restricted to the supervisor,” this 
simply means that only programs with access to the NI supervisor area can per¬ 
form the described operation or use the described feature. 

hi general, it is the responsibility of the operating system to make sure that user 
programs don’t have access to the NI supervisor area. Typically, this is done by 
using virtual address mapping to place the supervisor area in a memory region 
that user programs cannot access. 

Note: Some locations in the NI memory region don’t correspond to registers. 
The effect of reading or writing these locations is not defined, but is never of 
practical use to programmers. Typically, a Bus Error (see Section 1.4) is signaled. 
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1.3.1 For the Curious: The NI Base Address 

The physical base address of the entire NI region (both user and supervisor areas) 
is fixed at a value determined for each node by hardware. The actual physical 
address chosen by this method is the same for each node throughout the CM-5 
hardware. (Essentially, the physical address is set by two input pins on the NI 
chip, which are permanently wired either high or low for each circuit board). 

The NI region’s virtual base address, on the other hand, depends on the way the 
operating system sets up the virtual memory map. The operating system is free 
to map the NI memory regions to any virtual memory location, so long as both 
user and supervisor areas remain contiguous and user programs are prevented 
from accessing the supervisor area. 
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Figure 6. Sample virtual memory maps showing location of NI memory region. 
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The CMOST operating system distributed with the CM-5 maps the two NI re¬ 
gions into a contiguous 1024 Kbyte block, as described in the preceding section. 
Figure 6 shows two possible CMost virtual memory maps, one without the vec¬ 
tor units, and one with the vector units installed. 


1.3.2 NI Register Types 

There are three basic types of NI registers: 

FIFO Registers — These “registers” are actually the entry and exit points of 
send and receive FIFOs (First-In-First-Outs, or queues) associated with the 
CM-5 networks. Writing a value to a FIFO register pushes that value into the 
send FIFO of the corresponding network. Likewise, reading the value of a 
FIFO register pops a value from the receive FIFO of the network. 

Status Registers — These registers are composed of one-bit flags and multi¬ 
bit fields, which indicate the state of the NI and its message FIFOs. For 
example, most networks have two important status flags, send_ok and 
rec_ok, which indicate the current status of messages being sent or received. 

Control Registers — These are status registers containing flags that not only 
report the state of the NI, but also allow you to control it. Altering the value 
of a control register’s flags has a corresponding effect on the state of the NI. 
For example, each of the Control interfaces has one or more abstain flags that 
control whether or not the NI participates in the transactions of the network. 

The chapters of this manual that describe each of the networks also describe the 
NI registers that are associated with them, and describe the programming tools 
you can use to access these registers. 

Implementation Note: Some NI queue registers are mapped onto more than one 
memory location, and thus appear as regions of memory. Nevertheless, these re¬ 
gions of memory are still considered to be a single “register.” The specific 
memory location that you use in writing to these registers gives the NI additional 
information about the kinds of network transactions it should perform. (More on 
this in Section 2.3.2.) 

Performance Note: In terms of cycles, reading and writing NI registers is mid¬ 
way between reading the registers of the microprocessor and reading a value 
directly from processor memory (that is, not from cache memory). See Section 
8.1.1 for details on the time taken to read and write NI registers. 
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Important: Some registers are less than 32 bits long, even though they occupy 
a 32-bit memory location. When such a register is read, the value of the unused 
bits is undefined. However, when writing to the register, the unused bits should 
be written with either the same value that was last read from them, or with zeros. 
The effect of violating this restriction is not defined, but in some cases serious 
failures can result. (In at least one case, failing to zero out the unused bits causes 
your partition of nodes to crash. See Section 8.2.2.) 


1.3.3 NI Register and Field Names 

In this manual, the names of NI registers and register fields are given in the form 

nl_interface_purpose 

The interface part of the name identifies the network interface, and is typically 
one of the following abbreviations: 

dr Data Network (left and right) be broadcast interface 

ldr left interface com combine interface 

rdr right interface global global interface 

The purpose describes the purpose of the registrar or field. Some common exam¬ 
ples are 

send Register used to send a network message, 

reev Register used to receieve a message. 

send_ok Flag indicating that a message was sent successfully. 

rec_ok Flag indicating that a message has been received. 

For conciseness, this manual sometimes refers to a register or field by its purpose 
alone. However, this is done only when the intended reference is unambiguous. 

The appendixes of this manual include a memory map and a series of lists that 
exactly specify each register’s location and the position and length of any sub¬ 
fields it may have. 
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1.3.4 Nl Register and Field Programming Constants 

There are a number of predefined programming constants that you can use to 
refer to NI registers and fields in your code. 

These constants are defined in such a way that they can be used for both user and 
supervisor code; the names of the register and field constants are the same for 
both the user and supervisor areas, and are typically based on the names of the 
registers and fields themselves. 

To get access to these predefined constants, include the header file cmna.h: 

#include < cm/cmna.h> 

Note: Assembly-language coders may wish to load a more specific file of 
constants. See the discussion of the CMNA header files in Appendix H. 


Finding the Constant You Need 

Appendix A of this manual lists the names of the NI registers, fields, and flags, 
and gives the corresponding constants to use in accessing them. Appendix D pro¬ 
vides a complete list of the available low-level register and field constants. The 
types of predefined constants are described below. 


Register Constants 

The constants for registers specify the actual address of the register, and there is 
one such constant for each NI register. To get the name of the constant that corre¬ 
sponds to a register, uppercase the name of the register, and add the suffix “_a”. 
For example, the constant for the register ni_dr_status is NI_dr_status_a. 

Note for C Programmers: The register constants are unsigned pointer values. 
To use them in C code, you must cast them to type (unsigned *): 

unsigned *ni_dr_status = ((unsigned *) NI_DR_STATUS_A); 

If you don’t perform this casting step, the C compiler by default treats the 
constants as integers, causing warnings about “illegal pointer operations.” 
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Field Constants 

The constants for NI fields provide the starting bit position and length of each 
field. However, since a number of NI registers have some basic fields and flags 
in common, the name of the appropriate constant isn’t always directly derivable 
from the name of the field or flag in question. In many cases, you can obtain the 
constant name by uppercasing the field or flag name, and adding the suffix “_P” 
for the starting bit position, or “_l” for the field length. 

For example, the ni_dr_status register has a field named ni_dr_rec_tag. 
This field has two corresponding constants, ni_dr_rec_tag_p and 
ni_dr_rec_tag_Ii , that give, respectively, the position and length of the field. 

However, there is also a flag called ni_send_ok in the same register. Since most 
of the networks have a send_ok flag, there is a single pair of constants, named 
Nl_s end_ok_p and ni_send_ok_l, which apply to all the networks. 


NI Base Address Constant 

There is also a predefined constant that you can use to refer to the base address 
of the NI memory region (either user or supervisor) that you are using: 

NI_BASE — Base address of NI memory region (user or supervisor). 


1.3.5 C Macros Useful for Writing NI Code 

You can write NI code using any programming method that allows you to read 
and write memory addresses. However, the examples in this manual are written 
in the C programming language because there are a large number of existing C 
macros that you can use to streamline your code. These programming tools fall 
into two categories: 

■ Accessor macros that read or write the value of a specified register, flag, 
or field. (The send_ok and rec_ok macros are good examples.) 

■ Queue macros that take a number of arguments related to the sending of 
a single data value, and handle the necessary protocol for sending it. 

These tools are introduced individually in the chapters that follow, and there is 
a complete list of them in Appendix C. 
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Finding the C Macro You Need 

The predefined C macros typically have names based on the registers and fields 
they manipulate. For example, most network interfaces have an NI register 
named ni_interface_a tatus that contains the n±_interface_send_ok and 
ni_network_Tec_ok status flags. There is a single pair of macros, SEND_OK{) 
and REC_OK (), that is used to get the send_ok and rec_ok flag for any of the 
interfaces that have a ni_m£erface_s tatus register. 

Note: To get access to these predefined macros, your program must #include 
the header file cmna. h. (See Chapter 7 for more information.) 


1.4 Interrupts 

In addition to using registers to control the NI, you can also instruct the NI to 
signal an interrupt to the microprocessor under certain conditions, such as the 
arrival of a network message via a specific interface. These kinds of interrupts 
can be used to trigger calls to routines of your program (for example, handlers 
that automate the receipt of network messages). The NI also signals interrupts for 
fatal sofware/hardware errors and other important events. 

The NI can signal five different classes of interrupt: Red, Orange, Yellow, Green, 
and Bus Errors. Red interrupts and Bus Errors tend to be the most severe, and 
Green interrupts the least severe. 

The five interrupt classes can be briefly summarized as follows: 

■ Red interrupts indicate a hardware failure, or message checksum error. 

■ Orange interrupts indicate events that the operating system must handle. 

■ Yellow interrupts are triggered by fatal errors in user or OS software. 

■ Green interrupts are triggered by important non-fatal events that user or 
OS software may want to handle specially. 

■ Bus Errors indicate address errors in user or OS software that prevent a 
bus transaction from being completed. 

The five types of interrupts, along with the registers used for enabling and con¬ 
trolling diem, are described in more detail in Chapter 5. 
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In this manual, the names of interrupts are given as abbreviations based on the 
names of the register fields used to detect and clear them. For example, the Green 
interrupt that is triggered by the arrival of a broadcast message is: be rec ok. 


1.5 NI Reset 

Under certain conditions, the NI chip is completely reset. Among other things, 
this causes a number of its registers to be set to known states. The causes and 
effects of an NI Reset are described in Section 6.10. 


1.6 Using This Manual Effectively 

The first few chapters of this manual are mostly explanatory, describing the net¬ 
works of the CM-5 in detail and showing you how to use the NI programming 
tools associated with them. While these network-specific chapters present some 
brief code examples, none of these examples constitutes a complete NI program 
in and of itself. There’s a fair amount of information that you simply have to 
digest before a complete NI program makes sense. 

Beginning CM-5 programmers should read through the “generic” network de¬ 
scription in Chapter 2, and then read both of the network-specific chapters (3 and 
4), before turning to the complete sample program presented in Chapter 7. 

Experienced CM-5 programmers should read through Chapter 2 and skim chap¬ 
ters 3 and 4 to get a sense of how the networks operate, and then proceed to the 
sample program in Chapter 7 to see how NI programs are structured. 

Chapters 5 and 6 describe features of the NI that are primarily of interest to sys¬ 
tems programmers (things such as interrupts and other OS-related operations). 

Whatever your level of experience, read Chapter S. It presents a number of im¬ 
portant performance strategies and potential sources of progra mming errors that 
you should know about. 
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1.7 WARNING: Experiment at Your Own Risk 

In writing code that manipulates the NI chip, you are taking control of the lowest 
level of the CM-5’s hardware. That kind of power does not come without corre¬ 
sponding responsibilities and hazards. 

This manual sets strict protocols for reading and writing NI registers. When you 
use NI features in the manner described here, you should encounter no problems 
other than an occasional error message. 

If you step outside the bounds, however, the results can be as nasty as they are 
unpredictable. In some cases reading and writing NI registers incorrectly can 
even cause your partition of processing nodes to crash, potentially disrupting 
other timesharing users of the CM-5. 

So remember, if you choose to experiment with the NI, you have been warned! 
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Chapter 2 

A Generic Network Interface 


Each network interface of the Data and Control Networks has a corresponding 
register interface — a set of NI registers that are used to send and receive mes¬ 
sages through the network. These register interfaces typically have a number of 
features in common. This chapter presents a “generic” network interface that de¬ 
scribes these common features. With one exception (the global interface), all 
network interfaces conform to the model described here — individual variations 
for each network interface are discuessed in subsequent chapters. 

Important: The interface presented in this chapter is an abstract description. 
There is no actual “generic network interface” for the NI chip — merely a set of 
similar but independent network interfaces. 


2.1 Network Interface Registers 


For each interface that follows the generic model, the following NI registers are 
used to communicate with that interface: 


n±_interface_aend_£ ir s t 
ni_interface_aend 
n±_interface_Tecv 
nL_interface_atatua 
n±_interface_c ont r o 1 
ni_interface_pr ivate 


Used to send first value of a message. 
Used to send the rest of the message. 
Used to receive a message. 

Status register. 

Control register. 

Supervisor control register. 


The purpose and use of each of these registers and subfields is described in the 
sections below. Figure 7 (on the next page) contains a memory map showing the 
relative locations of these registers in the user and supervisor memory areas. 
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Figure 7. NI registers associated with each interface. 


.2 Network Messages 

A network message is a sequence of word-length (32-bit) values. Its content, for¬ 
mat, and length limit depend on the network. Each message is accompanied by 
a small amount of auxiliary information (such as the length of the message, a tag 
field, etc.). The format of this auxiliary data is also network-dependent. 

Sending a message involves writing its sequence of values to the send FIFO regis¬ 
ter of a network interface. As the message is written, the individual values are 
collected in the send FIFO. When the entire message has been written to the FIFO, 
the NI begins trying to send the message through the network. Similarly, receiv¬ 
ing a message involves reading its values from the receive FIFO register of the 
network interface. 


NI Version 2.2 (CM-5E), June 1994 
Copyright © 1994 Thinking Machines Corporation 














Chapter 2. A Generic Network Interface 


When a message arrives from one of the networks, the NT accumulates the mes¬ 
sage in the corresponding receive FIFO. When the entire message has been 
received, the NI sets a status flag, indicating a message is available. Your pro¬ 
gram can then read the individual words of the message from the receive FIFO. 

The send and receive FIFOs have a length limit (typically 5 words in the current 
implementation). Longer messages must be divided into packets at the sending 
node and combined at the receiving node. If you attempt to send a message that 
is longer than the total length of the FIFO (that is, a message that couldn’t possi¬ 
bly fit, even if the FIFO was empty) a Bus Error is signaled. 


2.2.1 Performance Note — Using Doubleword Operations 

You can use doubleword (64-bit) operations to read and write FIFO registers. A 
doubleword read or write has exactly the same effect as the corresponding pair 
of single-word (32-bit) reads or writes, but the doubleword operation is usually 
more efficient. (See Section 8.1.2.) From here on, where this manual refers to a 
“value” of a message, you should understand this as referring to either a single- 
or doubleword value. Any network-specific restrictions that prevent the use of 
doubleword operations are noted in the descriptions of the networks themselves. 


2.3 Sending a Message 

For each network interface, there is a single send FIFO, but two FIFO registers 
are used to access it in the process of sending a message: 

ni_interfac e_send_f .irst Used for first value of a message. 
ni_interface_send Used for the rest of the message. 

Important: There is a specific protocol to follow in sending a message: 

■ The first value of a message must be written to the send_f irst FIFO 
register. This tells the NI that a message is being composed, and also speci¬ 
fies the message’s auxiliary information (see Section 2.3.2 below). 

■ The remaining values (if any) must be written to the send FIFO register. 

If this protocol is not followed, a Bus Error is signaled, and the message currently 
being composed is discarded. 
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2.3.1 Message Discarding 

A message being written to the send FIFO register of a network interface can be 
discarded for any of a number of reasons: 

• The send FIFO may be temporarily full. 

* The supervisor may have disabled message sending for that interface. 

■ The message may not have been written according to protocol. 

Whatever the reason, when a message is discarded, it is completely discarded. 
Any previously written values for that message are removed from the send FIFO, 
and a new message can be started by writing a value to the send_f irst register. 
It is as though you never began writing the discarded message in the first place. 
(Writing additional values to the send register after a message has been dis¬ 
carded is legal, but has no effect.) 

Performance Note: You can use message discarding to your advantage and 
thereby make your code more efficient. Rather than check the send_ok flag af¬ 
ter writing each word of a message to the send FIFO, you can simply check the 
flag once, after the entire message has been written. (For more information, see 
Section 8.1.3.) 


2.3.2 Auxiliary information 

The auxiliary information of a message typically includes the length of the mes¬ 
sage in words, as well as network-specific data such as a message tag. This 
auxiliary information is transmitted implicitly when you write the first value of 
a message to the send_f irst register. 

The send_f irst register for each network interface is actually mapped onto a 
block of memory locations. Writing a value to any one of these locations has the 
effect of writing that value to the send_f irst register, but the actual memory 
location that you use implicitly supplies the auxiliary information of the mes¬ 
sage. (The low-order bits of the address actually contain the auxiliary data itself.) 

Another way of saying this is that the length of a message, among other thing s, 
determines the send_£irst address you must use to send it. 
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2.3.3 Calculating ni_/rrferface_send_first Addresses 


The send_f irst address for a network message is a 32-bit value of the form 


31 

_ t L. 

12 

14 12 

11 3 

0 

■8 


interface 

_ l _I_ 

y) auxiliary data 

□EX3 


where interface is the interface number (an integer from 0 to 7 representing the 
interface being used), auxiliary data is the auxiliary information of the message, 
and base address is the base address of the NI memory area (user or supervisor). 

The interface numbering is as follows: 

1 — Data Network (left and right) 3 — broadcast interface 

6 — left Data Network interface 4 — supervisor broadcast interface 

7 — right Data Network interface 5 — combine interface 

(The global interface does not conform to the generic interface model, so it does 

not play a part in this numbering scheme. The values 0, 2, and 4 are reserved.) 

The auxiliary data depends on the message, and each interface has its own format 
for this field. However, all the interfaces have at least one field in common: a 
length field, representing the length of the message in words. This field occupies 
the low-order 4 bits of the auxiliary data field (bits 3-6 inclusive). 

For the Curious: The auxiliary data is left-shifted three bits to leave sufficient 
space between send_f irst addresses for doubleword read/write operations. 
(See Section 2.2.1.) 


Send First Address Constants 

The following constants are used to construct send_f irst addresses: 

NI_base The NI base address. 

sf_fifo_offset The interface field offset (12). 

AUXILIARY_START_P The auxiliary data field offset (3). 

To construct a send_f irst address, combine the following values, left-shifted 
as shown: 

The NI base address: NI_BASE + 

The interface number: interface_number « sf_fifo_offset 

The auxiliary data field: auxiliary jiata « AUXILIAS.Y_STAR.T_P 
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The following interface jiumber constants are defined: 


DATA_ROUTER_FIFO 
LEFT_DR_FIFO 
RIGHTJDRJFIFO 
OSER_BC_FIFO 
SOTERVISOR_BC_FIFO 
COMBINE FIFO 


Data Network interface (1). 

Left Data Network interface (6). 

Right Data Network interface (7). 

User broadcast (BC) interface (3). 
Supervisor broadcast (SBC) interface (4). 
Combine (COM) interface (5). 


The interface-specific constants defining the auxiliary data field format are 
described together with the corresponding network interfaces in later chapters. 

For C Programmers: Appendix F of this manual includes examples of simple 
C macros that construct send first addresses for each network interface. 


2.3.4 C Macros for Writing a Message 

If you are programming in C, there are macros that you can use to automatically 
calculate the appropriate addresses for a message. For each interface, there are 
two send_£irst macros: 

am&_interface_BGn &_£irst ( auxiliary-info, value) 

CMNh_interface_B end_f irst_double (auxiliary-info, value) 

These are used to write the first value of a message to the send._f irst register. 
The only difference between them is that the send_f irst macro writes an 
unsigned value, while send_f irst_double writes a double. However, for 
these macros it’s not the type of data being sent that’s important, only the length. 

The send_f irst macro is intended to be used for sending word-length data, 
and the send_£irst_double macro is intended for sending doubleword data. 
In each case, you should coerce the values you send to the appropriate data type. 
For example, to send a data value of type float, you must first cast it as an 
unsigned value. To send a negative integer value, you must also first coerce it 
to an unsigned value. 

Performance Note: There are two kinds of send__£irst macros so that you can 
use doubleword register operations to make your code more efficient. (See Sec¬ 
tion 8.1.2 for more information.) For the most part, however, this manual focuses 
on singleword operations for clarity. 
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For the second and succeeding values of a message there is a different group of 
macros. For each network interface there are three macros that write values to 
the send register, one for each of the three data types you can send: 

C3XSA_inteTface_Bend_yoxd {value) 

CMNA._inteTface_8en&_£ loat {value) 
caJNA_inferyace_send_double {value) 

The sendjword macro writes an unsigned word-length value, and the other 
two macros write values of the indicated data types. Here there are three macros 
to allow you to send values of differing data types without having to coerce them. 
You’re not restricted to using only one data type, of course; you can use any com¬ 
bination of send_jype macro calls when sending a message. 

Important: Remember that the aend_type macros do not work unless they are 
preceded by a send_first or send_f irst_double call for the same network. 
You’ll get an error if you attempt to use them to send the first value of a message. 
If you have only one value to send, use the appropriate send_f irst macro. 


2.4 Receiving a Message 

For each network interface, the following register is used to receive messages: 

n±_interface_x ecv FIFO register from which values are read. 

A message is received by reading its value(s) in order from the recv register, one 
at a time. 


2.4.1 C Macros for Reading a Message 

Just as there are C macros for writing network messages, there are macros for 
reading them: three network-specific macros, one for each network interface: 

int value = CMNA_Jnterftzce_receive_word () ; 
int value = CtSHtA_interface_xece ive__f loat {) ; 
int value = CtSSK_interface_x eceive_doub 1 e () ; 

As with the send Jype macros, you are not restricted to reading values of a par¬ 
ticular type. You can use any combination of the rec _type in reading a message. 
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2.4.2 Detecting Arrival of a Message 

When a message arrives in the receive FIFO, the NI sets the rec_ok flag in the 
status register (see Section 2.5). You can repeatedly test the rec_ok flag to 
determine whether a message has arrived (for example, in a top-level loop). 

Alternatively, you can set a flag in the “private” register (See Section 2.7.) that 
causes the NI to signal an interrupt whenever the rec_ok flag is set. You can use 
this feature to “automate” message reception by having the interrupt trigger an 
appropriate message-reading routine in your program. 

Note: Access to the “private” register is restricted to the supervisor area. User 
programs, which do not have supervisor access, must make a system call to set 
the receive interrupt flag. 


2.4.3 Simulating the Arrival of a Message 

The supervisor has the additional ability to write a value to the recv register; this 
pushes a value into the tail end of the FIFO, as if it had arrived from the network. 
The supervisor can use this method to simulate the arrival of a message from the 
network (for example, when restoring the networks after a context switch), by 
writing the values of the message to the recv register in the same order as they 
are to be read out. (An appropriate value should also be written to the status 
register to provide the corresponding auxiliary information.) 

Note: An error is signaled if a value is written to the recv register when the 
receive FIFO is full (that is, when the ni_rec_f ull flag in the private register 
is set to 1 — see Section 2.7.5). 

Implementation Restriction: Currently, writing to the recv register does not 
work. The workaround for this restriction is for the node involved to send a mes¬ 
sage to itself — this message will wind up at the end of the receive FIFO, as if 
it had been written directly to the recv register. 
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2.5 The Status Register 


The ni_interface_.sta.tvLs register can be used to check on the progress of a 
message that is being sent, to detect when a message has been received, and to 
retrieve information about a received message. The status register includes the 
following flags and fields, which are the same for each of the network interfaces: 


n±_interface_statua 
ni_send_ok 
ni_send_space 
ni_s end_emp ty 
ni__rec_ok 
ni_rec_length 
ni_rec_length_lef t 


Status register. 

Flag, status of message being sent. 
Field, space left in send FIFO. 

Rag, indicates empty send FIFO. 

Rag, indicates arrival of a message. 
Field, total length of received message. 
Field, words left in receive FIFO. 


Note: The rec status fields always reflect the “current” message in the receive 
FIFO — the message that includes the next word waiting to be read from the 
receive FIFO. If there is no pending message, the fields are undefined. 


2.5.1 The “Send OK” Flag 

If the send FIFO becomes full, all attempts to write a message (either to start or 
to continue one) cause the message currently being composed to be discarded. 
You can tell that a message has been discarded by examining the send_ok flag. 

When the first value of a message is written to the send_f irst register, the 
send_ok flag is set to 1. As long as the message has not been discarded, this flag 
remains 1, indicating that the message is still being accepted. If the send_ok flag 
is still 1 after you have written the final value of a message, you can assume that 
that message has been accepted for delivery, and that you can start writing the 
next one. If the message is discarded, the send_ok flag is set to 0, indicating that 
the message has not been sent, and you should try resending the entire message. 
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2.5.2 The “Send Space” Field and “Send Empty” Flag 

The send_space field contains an estimate of the total space (in 32-bit words) 
left in the FIFO. The actual space remaining may be less; ni_send_space is 
usually correct, but may become invalid because of supervisor activity (such as 
when processes are swapped in and out). User code should not assume that push¬ 
ing a message shorter than this value is always successful. The send_empty flag 
is 1 whenever the send FIFO is empty — that is, when there is no pending mes¬ 
sage in the FIFO. 

Programming Note: NI programmers typically write an entire message to the 
send FIFO and then check the send_ok flag to see whether it was accepted, so 
the send_space field and send_empty flag typically aren’t used. 


2.5.3 The “Receive OK” Flag and “Receive Length” Fields 

Whenever a message is pending in the receive FIFO, the rec_ok flag is set to 
1, and remains 1 while any part of the message remains to be read from the FIFO. 
When no messages are waiting to be read, the flag is set to 0. (Attempting to read 
from the FIFO when rec_ok is 0 signals a Bus Error.) 

The ni_r e c_l ength_le £ t field contains the number of words of the current 
message that are left in the receive FIFO. You can assume that it is safe to read 
this many words from the receive FIFO. If you need the message’s original 
length, the ni_rec_length field always contains the total length (in words) of 
the current message as it was when it was received. 


2.5.4 Reading the Status Register Fields 

The general method for reading the value of an ni_interface_Bta.tvB field or 
flag is to read the value of the entire status register, and then extract the required 
fields from that value. (This cuts down the overhead of repeatedly reading the 
value of the register.) 

For each network, there is a C macro that returns the status register’s value: 
int value = CMNA._interface_8ta.tn8 () 
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Because the position and size of status fields and flags are the same for most of 
the network interfaces, there is a single set of macros that extract the status fields 
from the value returned by CMNA_imerface_s tatus: 


SEND_ok { status ) 

SEND_SPACE { status) 
SEND_EMPTY ( status) 
RECEIVE OK (Status) 


Gets send_ok flag from status value. 
Gets send_space field. 

Gets send_empty flag. 

Gets rec_ok flag. 


RECEIVE_LENGTH ( Status) 
RECEIVE LENGTH LEFT {status) 


Gets rec_length field. 

Gets rec_lengtli_lef t field. 


Note: A change in the broadcast interfaces requires the use of a different macro 
to access the rec_length_lef t field. See Section 4.1.6 for more information. 

For example, to get the three send fields from the broadcast interface status reg¬ 
ister, you could use the following C code: 


int value = CMNA_bc_status{); 

int send_ok = SEND_OK(value); 

int space_left = SEND_SPACE(value); 

int send_queue_empty = SEND_EMPTY(value); 

And to get the rec fields from the right data interface status register, you could 
use the following code: 


int value = CMNA_RDR_status(); 
int rec_ok = RECEIVE_OK(value); 
int message_length = RECEIVE_LENGTH(value); 
int words to_go = RECEIVE_LENGTH_LEFT(value); 


2.6 Abstaining from an interface — The Control Register 

Each of the Control Network interfaces has a control register, containing either 
one or two abstain flags. The names of the register and abstain flag(s) are: 

n±_interface_contxol Control register. 

ni_rec_abstain Normal receive abstain flag. 

ni_reduce_rec_abstain Combine reduction abstain flag. 

Note: The global interface, always the exception, uses a different name for this 
register. See Section 4.3 for more information. 
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2.6.1 Effect of Abstain Flags 

The rec_abs tain flag, when set to 1, causes the NI to “abstain” from receiving 
messages via the corresponding interface. That is, the NI does everything neces¬ 
sary to ignore the interface’s transactions: 

■ Arriving messages are simply ignored — they “disappear” with no indica¬ 
tion of their arrival, and the rec_ok flag remains 0. 

■ Messages that require the participation of every node (global synch, etc.) 
are allowed to complete without the abstaining node’s participation. 

■ Messages that require a value (scan messages, for example) are effectively 
given an appropriate identity value for the type of message being sent. 

While the rec_abstain flag is set for a given interface, it is an error to try to 
said a message via that interface from the abstaining node. Attempts to write the 
send__f irst or send registers under these circumstances signals a Bus Error. 


2.6.2 Combine interface Abstain Flags 

The ni_reduce_rec_abstain flag is only defined for the combine interface, 
and only applies to reduction operations. 

In addition, reduction operations treat the value of the rec_abstain flag differ¬ 
ently from all other interface operations. 

For more information, see Section 4.2.10. 


2.6.3 Reading and Writing the Abstain Flag 

To read and write the the abstain flag of a network, you can use these C macros: 

value = CMNA_read_abstaln_f lag ( register) ; 

CMNA_write_abstain_£lag ( register, value) ; 

The register argument is a register address constant, which is defined separately 
for each network. 
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2.6.4 Use the Abstain Flags Safely 

The abstain flag for a given interface should only be changed when that interface 
is not in use. Specifically, when a interface’s abstain flag is changed, 

■ The said FIFO must be empty (that is, the send_empty flag must be 1). 

■ The receive FIFO must be empty (the rec_ok flag must be 0). 

■ There must be no messages in transit via that interface. (There is no flag 
to detect this; your program must simply be written so that this is the case.) 

The effects of c hang ing a interface’s abstain flags while the interface is in use are 
unpredictable — your code may produce erroneous results, or signal an error. 

This restriction generally requires that you use one of the interfaces (for example, 
the global interface) to synchronize the nodes and halt the operations of another 
interface while you change that interface’s abstain flags. For this reason, most NI 
programmers set the abstain flags once, at the beginning of a program or routine, 
and then leave them set that way until the program or routine finishes executing, 
changing the flags within the routine only where absolutely necessary. 


2.6.5 Being a Good Neighbor 

Important: Some programming systems (such as CMMD) use the abstain flags 
for their own purposes. These systems are written with the assumption that the 
abstain flags do not change unexpectedly, and if the flags do change these sys¬ 
tems may not operate correctly. 

When you alter the values of the abstain flags, you must take care to save the 
original settings of these flags and to restore them before handing control back 
to these systems. Failing to do so can cause either user or OS code to signal ob¬ 
scure errors that are hard to trace. 
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2.7 The Private Register 


Each of the interfaces also has a “private” control register, containing a number 
of control flags and status fields for supervisor operations. Most of these sub¬ 
fields are interface-dependent; the few that axe not are: 


ni_interface_pz i va t e 
ni_rec_ok_ie 
ni_lock 
ni_rec_stop 
ni_send_s top 
ni rec full 


Private register. 

Flag, “Receive OK” interrupt enable. 
Interface lock flag. 

Interface stop flag (except Broadcast intf.). 
Interface stop flag (Broadcast intf. only). 
Flag, indicates receive FIFO is full. 


The broadcast interface has one exception: the ni_rec_stop flag is not defined; 
in its place is a flag called ni_se»d_stop, which operates differently. (See Sec¬ 
tion 2.7.4.) 


Usage Note: The private register is accessible only from the supervisor area; 
users without supervisor access must make a system call to change the flags in 
this register. 


2.7.1 Message Receipt interrupts — The Rec interrupt Enable Flag 

When the ni_rec_ok_ie flag is set to 1, a Green interrupt is signaled whenever 
a new message becomes available at the front of the interface’s receive FIFO (in 
other words, whenever the rec_ok status flag is set to 1 for a new message). 

A message may become available either by arriving from the network into an 
empty FIFO, or by being the next message in the FIFO when the last word of the 
current message is read out. A different Green interrupt is signaled for each net¬ 
work interface, and the interrupt for each interface can be independently enabled 
and disabled by setting the rec_ok_±e flag for the interface. 

The Green interrupts that can be signaled are: 

dr rec ok ldr rec ok rdr rec ok 

be rec ok sbe rec ok com rec ok 

For more information about these interrupts, and about interrupts in general, see 
Section 5.1. 
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2.7.2 Clearing the Interface’s Send FIFO — The Lock Flag 

The supervisor can use the ni_lock flag to temporarily “lock” the interface — 
that is, prevent use of the interface in a way that is transparent to a user program. 

The lock flag is normally 0. When it is set to 1, the following effects occur: 

■ Any message currently being written to the send FIFO is discarded. 

■ The send_ok flag is set to 0 and remains 0 — even if you attempt to write 
a new message to the send FIFO. 

■ The value of the ni_interface_a-pa.ee field is set to 0 and remains 0. 

In other words, setting the lock flag to 1 clears the send FIFO, and then makes 
it seem as if the FIFO is permanently full. 


2.7.3 Grabbing the Receive FIFO Registers — The Rec Stop Flag 

The supervisor can temporarily grab control of a interface’s receive FIFO and 
status register by setting the interface’s ni_rec_stop flag. Since this involves 
the joint cooperation of the microprocessor and the NI, a special request/grant 
protocol is used. Specifically, 

■ The microprocessor writes a 1 to the interface’s rec_stop flag, indicat¬ 
ing it wants direct control of the reev and status registers. (Note: The 
rec_stop flag is not changed to 1 until the stop operation is completed.) 

■ If a message is currently arriving from the interface, the NI finishes receiv¬ 
ing the message and stores it in the receive FIFO. 

■ The NI then stops receiving messages from the interface, and finally sets 
the rec_stop flag to 1, indicating that the stop operation is completed. 

Once the rec_stop flag is set, the supervisor may freely read and write the val¬ 
ues of the reev and status registers (for example, to push additional messages 
into the FIFO, or to clear the FIFO altogether). When the supervisor is finished 
with the reev and status registers, writing a 0 to the interface’s rec_stop 
flag restores normal network operations. 

Important: It is an error for the supervisor to attempt to write values to the reev 
and status registers while the stop flag is 0. The effect of doing so is unde¬ 
fined, but is not likely to be pleasant. 
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2.7.4 Blocking Unsent Broadcast Messages — The Send Stop Flag 

The broadcast interface does not have a rec_stop flag. Instead, the same posi¬ 
tion in the private register is used for a flag called ni_send_stop, which has 
a different purpose. When the send_stop bit is set, it prevents any complete 
messages waiting in the broadcast send FIFO from being sent over the network. 
This mechanism is mainl y used by the supervisor during process swaps, to hold 
messages in the interface send FIFO until they can be safely removed and saved. 


2.7.5 Detecting a Full Receive FIFO — The Receive Full Flag 

The ni_ree_full flag, when set, indicates that the interface’s receive FIFO is 
full. This is critical to network performance; if too many nodes have full receive 
FIFOs, the network can become clogged with unreceived messages, and this can 
prevent new messages from being delivered to their destinations — even if the 
destination nodes actually have sufficient space in their receive FIFOs. 


2.8 Using a Generic Network interface 

To sum up, the strategy to use in accessing a network interface’s registers is: 

■ To send a message, write the first word to the send_first register, and 
any remaining words to the send register. 

■ Check the send_ok flag to see if the message was discarded, and if so, 
retry sending the entire message. 

■ To receive a message, check the rec_ok flag to see if a message is in the 
FIFO, and if so, use the length and lengthJLef t fields to determine the 
number of words to read from the recv register. 

■ Use the remaining fields of the status register to obtain other interface- 
specific information about the state of the send and receive FIFOs. 

■ Use the abstain flag(s) in the control register to cause individual 
nodes to ignore the transactions of the interface. 

■ Use the private fields and flags for supervisor features such as disabling 
send FIFOs, checking for full receive FIFOs, and setting interrupts. 


32 


NI Version 2.2 (CM-5E), June 1994 
Copyright © 1994 Thinking Machines Corporation 



Chapter 2. A Generic Network Interface 


2.9 From the Generic to the Specific 

The interface described in this chapter is an idealized view of a network inter¬ 
face, lacking a specific purpose, a detailed description of message protocol, or 
network-related restrictions on usage of the interface registers. 

The next two chapters present a description of the Data Network and Control 
Network. These chapters present the purpose, protocol, and restrictions of each 
interface provided by the CM-5 networks, building on the generic interface 
description presented in this chapter. 
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The Data Network 



The Data Network consists of two halves, the left interface (LDR) and right inter¬ 
face (RDR). Each half of the network is connected to all nodes, and can be used 
independently. The two halves of the network can also be accessed together as 
the single Data Network (DR): 



Figure 8. The three interfaces of the Data Network: DR, LDR, and RDR. 


For each of these network interfaces there is a separate register interface. This 
chapter describes these register interfaces, and shows how to use them to send 
messages through the Data Network. 

Terminology Note: The network acronyms (DR, LDR, RDR) are a historical 
anachronism, and are retained in this manual only because the C constants used 
to access the Data Network still refer to the three interfaces by the old abbrevi¬ 
ations. In addition, the obsolete term “router” is occasionally still used in the 
progr ammin g contants to refer to the Data Network hardware. “Network” is cur¬ 
rently preferred, as a more generic and thereby more accurate descriptive term. 
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3.1 The Data Network Register Interfaces 

The three Data Network interfaces are based on the generic model presented in 
Chapter 2. There are three sets of interface registers: one for each half of the 
network (LDR and RDR), and one for the combined network (DR). Each network 
interface is used to send and receive messages, with the following conditions: 

■ Sending a message via the DR actually sends it by either LDR or RDR, 
depending on the load of the two interfaces. 

■ The DR interface cannot be used to receive any messages, and is mutually 
exclusive with the two half-network interfaces. In other words: 

■ Writing a message to the DR send FIFO excludes using either the 
LDR or RDR at the same time. Likewise, writing a message to either 
the LDR or RDR send FIFOs excludes using the DR interface. 

* While a message is being sent, any excluded interface(s) remain ex¬ 
cluded until the message has been written and accepted for delivery 
by the network. Also, the status register(s) of excluded interface(s) 
are invalidated and should not be used. 

■ The two half-network interfaces are not mutually exclusive, and in fact 
can be used simultaneously. In other words, network messages can be sort 
and received concurrently via both the LDR and RDR. 

For each Data Network interface, the following registers are used: 

nl_dinterface_send_£ix8t Used to send the first value of a message. 

n±_dinterface_a end_f irst_long Used for first value of long message. 

ni_dinterface_aend Used to send the rest of the message. 

ni_dinterface_xecv Used to receive a message. 

n±_dinterface_Bta.tvLB Status register. 

ni_dinterface_atatua_long Status register for long messages. 

ni_dinterface_BtsLtua_a.il Alternate status register. 

ni_dinterface_8tatua_pop Status reg, also receives messages. 

ni_dinterface_px ivate Supervisor control register. 

The dinterface part of these names is a unique abbreviation for each interface: 

dr - Data Network ldz - left interface rdr - right interface 
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Figure 9 is a memory map indicating the relative locations of these registers in 
the user and supervisor areas. 


The Data Network Registers at a Glance: 

hex offsets 

dr ldr rdr 
- - - 

„ n:L ~ X ~ aend ~ £lrBt ~ :1 ' Qng 0x8000 0x100000x18000 

ni x send first 

--- 0x1000 0X6000 0x7000 


ni_x_status_long 
ni_x_s tatus_al 1 
nl_x_sfcateu a p op 
ni_x_aen.d 
nl x rec 


0x027 0 0x0C70 0XOE7 0 
0x0260 0x0060 0X0E60 
0x0250 0X0C50 0x0E50 

- 0x0040 0x0E40 

0x0230 0X0C30 OxOE30 
- 0X0C20 0X0E20 


ni_x_private 
nl x status 


0x0208 0X0C08 0x0E08 
0x0200 0X0C00 OxOEOO 


ni_interface_send_first Addressing Patterns 


user/supervisor bit addressing mode 

j interface t 

NI base address f index f 

R - X 0000 0 0 1 | X tag 

)R - X 0000 1 I 1 I 0 X I tag 


- X 0000 1 | 1 I 1 X I 

20 1 19'l8 15 1 14 12*11 10 


tag 

length 

000 

tag 

length 

! 000 

tag 

length 

! 000 


niJnterface_send_firstJong Addressing Patterns 


user/supervisor bit 
HZ base address \ 


interface 

index 



FT 

0 

Gu 

0 

0 


addressing mode 

i length 

£ length 


length 


16 15 12 11 


[ 000 | 

3 1 2 o' 


Figure 9. Nl registers associated with each of the Data Network interfaces. 


NI Version 2.2 (CM-5E), June 1994 

Copyright © 1994 Thinking Machines Corporation 











NI Programmer’s Handbook 


The following related registers are also used to control Data Network features: 


ni_longest_dr_message 

nl_hodgepodge 

n±_msg_too_long_ie 
ni_user_tag_mask 
ni_rec_interrupt_mask 
ni_user_rec_interrupt_mask 
ni_dr_me s sage__count 
ni cotint mask 


Length limit on Data Network messages. 
Register with “hodgepodge” of flags: 

Message too long interrupt enable. 
User/supervisor tag reservation register. 
Contains tag value interrupt flags. 
Contains tag value interrupt flags. 
Contains current message count. 
Contains tag-count enable flags. 


The propose and use of these registers are described in the sections below. 


3.2 Data Network Messages 

The Data Network is essentially asynchronous in operation — nodes can send 
and receive messages freely, so long as enough nodes are receiving messages so 
that the network does not become clogged (see Section 3.9). The destination 
node of a Data Network message is determined by an address word that is added 
to to the message as it is being written to the send FIFO. (Note: The address word 
is removed in transit. It does not count as a message word with reference to the 
length limits of the send and receive FIFOs.) 


3.2.1 Short and Long Data Network Messages 

Each of the three Data Network interfaces can send messages of two types: short 
and long. A short message is sent as described in Chapter 2, and has a length 
limit of 5 words. A long message is sent via an alternate register interface, and 
has a length limit of 18 words. The long message interface is intended for mes¬ 
sages that consist primarily of large quantities of data. 

Implementation Note: The long message feature of the Data Network is an 
addition as of Version 2.2 of the NI chip. The short message type is actually the 
same Data Network message format used in previous NI versions, and is retained 
in Version 2.2 for software compatibility reasons. 
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3.2.2 Long Data Network Message Interrupt 

The NI register n±_longest_dr_message overrides the default length limits 
for long messages — trying to send a message longer than the value in 
n±_longest_dr_message signals a Yellow interrupt (message too long). 
This is intended to provide compatibility in CMs that contain NT chips of differ¬ 
ent versions. The flag ni_msg_too_long_ie in the n±_hodgepodge register 
controls this interrupt feature. If ni_msg_too_long_ie is 1, the mes¬ 
sage too long interrupt is signaled. If n±_msg_too_long_ie is 0, no 
interrupt is signaled. In either case, however, a Bus Error is signaled. 


3.2.3 Data Network Message-Sending Conventions 

Data Network messages are atomic; individual messages are not sent through the 
network until all the words of each message have been written into the send 
FIFO, and arrival of each message is not reported until all the words of the mes¬ 
sage have arrived in the receive FIFO. The component words of a single Data 
Network message are always received in the same order as they were sent. How¬ 
ever, if you use multiple Data Network messages as “packets” to send long 
messages from one node to another, the order in which the packets arrive is not 
guaranteed to be the same as the order in which they were sent. 

Your code should not depend on having separate Data Network messages sent to 
the same node arrive in some predictable order. Instead, your code should in¬ 
clude data in the packets (for example, an offset into the original message) that 
allows the receiving node to arrange the packets into the correct order. 


3.3 Data Network Addressing 

The Data Network uses two kinds of addressing: physical and relative. Each 
node of the CM-5 has a unique physical address based on its location in the CM-5 
hardware. This represents an “absolute” address, giving the node’s location with 
respect to the entire machine. 

Each node also has a unique relative address based on its location in its partition. 
Relative addresses run from 0 (for the first node in the partition) up to one less 
than the total number of nodes in the partition. (See Figure 10.) 
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Figure 10. Relative addressing of nodes in a partition. 


You can get the address of the node executing your code, as well as the total 
number of nodes in the current partition, by examining these C variables: 

CMNA_sel£_address Address of current node. 

04 NA_partitlon_size Number of nodes in current partition. 

The values of these variables are automatically defined for each of the nodes. 
The value of CMNA_jpartition_size is also defined for the partition manager. 

Note: The partition manager is always located at an address outside the partition, 
and so does not occupy any of the relative addresses of the partition. (For more 
information, see Section 7.1.) 


3.3.1 Physical and Relative Addressing Modes 

Just as there are two kinds of addressing, there are also two “modes” of sending 
a Data Network message: physical and relative. The mode a message is sent in 
is determined by a mode flag in the auxiliary data of the message. 

When a message is sent in physical mode, its address word is treated as a physi¬ 
cal address, and the message can be sent anywhere wi thin the Data Network. 
(Only the supervisor is allowed to send messages in physical mode.) 

When a message is sent in relative mode, the address word is treated as a relative 
address, and is translated into a physical address based on the current partitioning 
arrangement. This translation is performed automatically by the NI hardware, 
using a chunk table , described in Section 6.3. The translation also includes auto¬ 
matic error checking to make certain that the supplied address is a legal relative 
address for the current partition. Messages that contain illegal relative addresses 
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are not sent through the network; instead, the sending NI signals a Yellow inter¬ 
rupt (bad relative address). 

For the Curious: The relative addresses in a partition are always contiguous — 
that is, there are no legal relative addresses in a partition that do not correspond 
to existing functional nodes. This is in contrast to physical addresses, which can 
contain gaps corresponding to nonfunctional nodes or to network locations that 
are not connected to actual CM-5 hardware. (See Section 6.3.) 


3.4 Sending and Receiving Messages 

The Data Network message format is the same for all three interfaces (and for 
short and long messages alike). The first word of the message is a 20-bit destina¬ 
tion address. The remaining words form the content of the message, which must 
be no longer than the length limit allowed by the message type in use. 


Address 


Dalai 


if' 


Data 2 


•** -■* •« • v; '' ' 


Dalai* 


Word 1 


n+1 


Figure 11. Data Network message format 


For short messages, the data length limit is currently 5 words, and is given by the 
constant MAX_rotjter_msg_words . For long messages the limit is 18 words. 
(The ni_longest_dx_message register value, if less, overrides these limits.) 

The auxiliary information of the message consists of the length of the message 
in words (excluding the address word), a 4-bit tag value, and an addressing mode 
flag that determines how the address word is interpreted. 

Important: The address word of the message must be zero-extended to 32 bits. 
Failure to ensure that the address word is zero-extended to the full 32 bits can 
trigger a serious error, even causing your partition to crash. 
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3.4.1 Sending Short Messages 

The protocol for sending a short message is as described in Chapter 2. The fol¬ 
lowing FIFO registers are used to send messages: 

ni_dinterface_B end_f ir s t Used for first value of a message. 

ni_dinterface_sen& Used for the rest of the message. 

and for each dinterface there are corresponding sendjf irst and send macros: 

cmK_dinterface_aend_t±xBt (tag, length, value) 
CKSK_dinterface_BBrxd_i ir st_double ( tag , length, value) 
C3XHtK_dinterface_BBnd_yoTd (value) 

CMNA_<imfer7i2ce_send_£loat (value) 

CMNh_dinterface_Bend_double (value) 

For the send__£irst macros, the length argument is the length of the message 
in words (excluding the address word), the tag argument is the message’s tag 
value, and value is the first value of the message. For the send macros, value is 
the second and succeeding values of the message. 

Note: Currently you are limited to using tag values from 0 to 7. All other tags 
are reserved for supervisor use. 


Auxiliary information for Short Messages 


The 9-bit auxiliary information field of a short message has the form 
8 0 


md\ 


tag 


length 


where 


md is the addressing mode (0 = relative, 1 = physical) 
tag is the 4-bit tag value 

length is the length of the message in words, excluding address word 


The following constants specify the starting bit positions of these fields: 


NI_DR_SEND_AtJXILIARY_ADDRESS_MODE_P The md field offset (8). 
NI_DR_S END_AtJXIL IARY_TAG_P The tag field offset (4) . 

ni_dr_send_auxiliary_length_p The length field offset (0). 
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To construct a send_£irst address, add the following values: 

The md flag: md « ni_dr_send_auxiliary_address_mode_p 

The tag value: tag « ni_dr_send_auxiliary_tag_p 

The length value: length « ni_dr_send_aoxili ar y_length_p 

The md flag is 0 for a message with a relative destination address, and 1 for a 
message with a physical destination address. The following constants can be 
used to specify the md flag value: 

relative Relative node addressing (0). 

physical Physical node addressing (1). 

Note: Sending messages with physical addresses is reserved for the supervisor. 
If user code tries to send a message with a md flag of 1, a Bus Error is signaled. 

The tag can be any value from 0 to 7 inclusive for user messages, or from 0 to 
15 for supervisor messages. Message tags are described in more detail in Section 
3.5.4 below. The length field can have any value from 1 up to max_rotjt- 
ER MSG WORDS. 


3.4.2 Sending Long Messages 

The protocol for sending a long message is the same that for short messages, 
except that the first word of the message must be written to a special register: 

n±_dinterface_3 end_f ir s t_long Used for first value of long message. 

and for each dinterface there are corresponding send_f irst_long macros: 

CMNA_dinterface_aend_f irst_long (tag, length, value) 
CMtJA_dinterface_a end_first_doub 1 e_long( tag , length, value) 


Send First Long Address Format 

The send_£irst_long address for a Data Network message is a 32-bit value 
of the form 


31 


19 


17 


15 


13 12 


0 


base addr 


0 0 


intf 


0 0 


auxiliary data 


0 0 0 
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where intf is the interface number (an integer from 0 to 3 representing the Data 
Network interface being used), and auxiliary data is the auxiliary information. 
The following intf values are defined: 

0 - Not used 2 - LDR network interface 

1 - DR network interface 3 - RDR network interface 


Auxiliary Information for Long Messages 


The format of the auxiliary information is 
9 4 


md 


length 


tag 


where 


md is the addressing mode (0 = relative, 1 = physical). 

length is the length of the message in words, excluding address word. 

tag is the 4-bit tag value. 


Aside from size and position, these three fields are the same as those defined 
above for the auxiliary information of a short message. 


3.4.3 Receiving Messages 

For each interface, the following register is used to receive messages: 

ni_dinterface_xecv FIFO register from which values are read. 

Both long and short messages are received as described in Chapter 2, by reading 
successive words of the message from the recv register. (Messages can also be 
received via the ni_dinterface_Bta.tv.s_po-p register. See Section 3.5.3.) 

To receive a message from the LDR or RDR, use the network-specific reading 
operations described in Section 2.4.1: 

value = CMt>tA_dinte?face_receive_yoTd O ; 
value = CtmA_dinterface_xeceive_float () ; 
value = CMNA_dinterface_Teceive_doxsble 0 ; 
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Important: There are no message-receiving macros for the DR. You must use 
the LDR and RDR to receive messages sent via the DR — the DR interface cannot 
be used to receive messages. 

Supervisor Usage Note: Currently, a hardware defect in the NI chip does not 
allow the Data Network recv registers to be written by the supervisor to simu¬ 
late the arrival of messages, etc. The workaround is for a node to send a message 
into the network using its own address as the destination. Ass uming the network 
is clear (as it is, for example, during context switches) this causes the message 
to be delivered to the front of the node’s receive queue. 


3.5 The Status Registers 


3.5.1 The Standard Status Registers 


Each of the Data Network interfaces has two main status registers, one each for 
short and long messages, which contain the subfields shown below: 

nl_dinteTface_Bta.tus_long Status register for long messages. 
ni_dinterface_3 1 atus Status register for regular messages. 


ni_send_ok 

n±_send_space 

ni_rec_ok 

ni_rec_length 

ni_r ec_length_le£ t 

ni_dr_rec_tag 

n±_dx_send_s t at e 

ni_dr_rec_a tat e 

ni_router_done_complete 


Flag, status of message being sent. 
Field, space left in send FIFO. 
Flag, indicates receipt of message. 
Field, total length of message. 
Field, words left in the FIFO. 

Field, tag value of the message. 
Field, status of send FIFOs. 

Field, status of receive FIFOs. 
Rag, indicates empty send FIFOs. 


The only difference between the status and status_long registers is that in 
the status__long register the send_space, rec_length, and 
ree_length_lef t fields are five bits long instead of four to accommodate the 
extra length of long messages. 


Each of these fields has the same value in both the status and status_long 
registers, except where the value exceeds 15. In this case, the status_long 
field contains the correct value, while the status field is always 15. If the super¬ 
visor writes a value to any of the four-bit status fields, the corresponding 
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five-bit statusJLong field is automatically updated to the same value, with a 
0 for the most significant bit. The reverse is also true, with the status field 
being set to 15 as described above if the status_long value exceeds 15. 

The macros used to get the rxl_interface_e tatus value for each interface are: 

int value » CMNA_dr__send_status () ; 
int value = CMNA_1dr_status(); 
int value = CMNA_rdr_status(); 

The send_ok, send_space, rec_ok, rec_length, and recJLengthJLef t 
subfields are as described in Chapter 2. The dr_rec_tag field is described in 
Section 3.5.4 below, the dr_{ send, rec }_state fields in Section 3.5.5, and the 
ni_router_done_complete flag is described in Section 3.5.6. 

Implementation Note: The subfields ni_dr_send_state and 
n±_dr_rec_state, and the flag ni_router_done__eonrplete apply to all 
three interfaces. They are accessible only from the DR interface (that is, their 
values are defined only for the ni_dr_status register). 


3.5.2 The “Status All” Alternate Status Register 

Each Data Network interface also has an alternate status register, which gathers 
information about all three Data Network interfaces into a single word value: 

nl_dinterface_eta.tvLe_a.il Alternate status register. 

nl _ftrstintf_x&c_ok Flag, indicates receipt of message. 

nl_secondintf_xec_ok. Flag, receipt of other interface message. 

ni_dinterface_een&_o\t Flag, send OK flag of interface, 

ni _firstintf_xec_tag Field, tag value of LDR message. 

ni_secondintf_xec_tag Field, tag value of RDR message, 

ni _firstintf_xec_l ength_long Field, total length of LDR message. 

nl_secondintf_x ec_length_long Field, total length of RDR message. 
ni_dinteTface_eend_epace Field, space left in DR send FIFO, 

ni _firstintf_xec_&ll_± all_down Flag, indicates All Fall Down message. 
ai_secondintf_xec_&ll_£&ll_dovn Flag, indicates All Fall Down message. 
ni_router_done_complete Flag, indicates empty send FIFOs. 

Note: Currently, there are no predefined C access routines for the status_all 
register; you must use the predefined register address constants. 
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In the field names listed above, the firstintf and secondintf portions of the names 
are different for each network interface: 


Interface 

dinterface 

firstintf 

secondintf 

DR 

dr 

ldr 

rdr 

LDR 

ldr 

ldr 

rdr 

RDR 

rdz 

rdr 

ldr 


In general, firstintf is the same as dinterface, while secondintf is the opposite 
interface in the pair of LDR and RDR. This is so that when a program is using 
the two half networks, the status values for the “current” and “opposite” halves 
of the network can be obtained from the same positions in the status_all reg¬ 
ister, regardless of the interface (LDR or RDR) that is in use. 

The flag and field values in the atatus_all register are copied from the 
appropriate Data Network status registers, with the exception of the 
rec_al l_f al l_down flags that are taken from the ni_dinterface_j?z ivata 
register (see Section 3.6). At all times, the value of the status_all register 
mirrors the current values available from the individual status registers. 


3.5.3 The “Status Pop” Register 

The status_all register also has a convenient doubleword alias: 

n.±_dinterface_B tatus_pop Status register, also receives messages. 

The status_pop register is identical to the status_all register, except that 
the status_pop register can only be read with a doubleword operation. 

When this is done, the first word of the result is the current value of the sta- 
tus_all register. The second word of the result is a value popped from the 
appropriate dinterface receive FIFO, if a value is available. 

Thus, a single doubleword read of the s tatus_pop register can be used to check 
whether a value is available for reading from the network interface, and also to 
get the value if there is one. 

Note: The s tatus_pop feature is defined only for the LDR and RDR interfaces, 
since it is not possible to read a value from the DR interface. 

Also, there is currently no ntl_dinterface_9 tatus_pop_a register constant; use 
the offset value (x40) shown on the NI memory map. 
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3.5.4 Message Tags 

The tag values of Data Network messages are used to distinguish between differ¬ 
ent types of Data Network messages. The status register field dr_rec_tag 
always contains the tag value that was sent with the current message. 

Tag values are not mandatory. You can, for instance, simply supply a tag value 
of 0 for all Data Network messages. 

Tag values are primarily used for 

* distinguishing between user and supervisor messages 

* causing interrupts to be signaled when messages are received 

* helping the NI determine when the Data Network is clear of user messages 

To get the rec_tag field, use the macro 
RECEIVE TAG (status) 


User/Supervisor Tag Reservation 

Some tag values are reserved for supervisor use, to distinguish between supervi¬ 
sor and user messages. The remaining tags can optionally be used in user 
programs to distinguish different types of user messages. 

The NI has a register that controls the reservation of tag values: 

ni_user_tag_mask User/supervisor tag reservation register. 

Only the low-order 16 bits of this register are used, one for each of the possible 
tag values (0 to 15). If the nth bit of the user_tag_mask register is 1, then tag 
value n is reserved for supervisor use. 

Since the tag_mask register is only accessible by the supervisor, it effectively 
acts as a set of permission switches, controlling which tags the supervisor allows 
user messages to have. If a user program attempts to send a message with a 
supervisor-reserved tag, a Bus Error is signaled. 


Tag Fields and interrupts 

Tag values can be used to trigger interrupts; when a message with an interrupting 
tag value becomes available for reading in the receive FIFO, the NI signals an 
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interrupt to the microprocessor. (A message becomes available either by arri vin g 
at an empty receive FIFO, or by being the next message in the FIFO when the 
current message is read out.) Tag value interrupts can be used to cause the micro¬ 
processor to execute a specific section of code whenever a message with an 
interrupting tag becomes available for reading. 

The following registers and register flags are used to determine which tag values 
cause interrupts, and how they are signaled: 

n±_rec_±nterrupt_mask Register, Supervisor tag interrupt flags. 
ni_user_rec_interrupt_mask Register, User tag interrupt flags. 
ni_hodgepodge Register containing “hodgepodge” of flags: 

ni_ldr_rec_tag_ie LDR supervisor tag interrupt enable. 

ni_rdr_rec_tag_ie RDR supervisor tag interrupt enable. 

n±_ldr_user_rec_tag_ie LDR user tag interrupt enable. 
ni_rdr_user_rec_tag_ie RDR user tag interrupt enable. 

The interruptjnask registers each contain 16 flags, one for each tag value. 
If the nth bit of either register is 1, it indicates that an arriving message with a 
tag value of n should signal an interrupt. However, the “supervisor” tag value 
register ni_rec_±nterrupt_mask has overriding contol over which tag val¬ 
ues signal interrupts. The ni_user__rec_ ii lnterr'upt__mask register is 
dependent on the value of ni__rec_±nterrupt_mask; only if the nth bit of the 
ni_rec_interrupt__mask is set to 0, will a 1 in the corresponding bit of 
ni_user_rec_interrupt_mask cause an interrupt to be si gnal ed. 

The interrupt enable flags in the ni_hodgepodge register enable and disable the 
interface-specific supervisor and user interrupts (see Figure 12). 

When a message with a tag value of n arrives at the LDR interface of the Data 
Network, the nth flag bit of ni_rec_±nterrupt_mask is checked. If the flag 
is 1, then a Green interrupt (ldr rac tag) is signaled. If the flag is 0, the nth 
flag bit of the ni_user_rec_i.nterrupt_mask register is checked. If this flag 
is 1, then a Green interrupt (ldr user rac tag) is signaled. If the flag is 0, 
then no LDR interrupt is signaled. A similar method is used to determine whether 
to signal the rdr rec tag and rdr user rac tag interrupts when a 
message arrives via the RDR interface. 

hi all cases, if the nth bit of n±_rec_interrupt_mask is 1, the arrival of a 
Data Network message with tag value n by either interface (LDR or RDR) 
always signals a Green interrupt (dr rec tag). 
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Figure 12. Tag value Interrupt paths for Data Network messages 


The ni_user_jcec_interrupt__mask register is both readable and writable by 
user programs, but the interrupt enable flags n±_ldr_user_rec_ie and 
ni_r dr_us er_r e c_i e are writable only by supervisor programs. The intent of 
this is to allow the supervisor to use the user interrupt enable flags as “permis¬ 
sion” bits — by setting either of the two user_rec_ie flags, the supervisor 
grants to user programs the ability to turn interrupts on and off for all tags not 
already reserved for supervisor interrupts. This avoids the need for a supervisor 
call whenever a user program wants to enable or disable user interrupts. 

The ni_rec_±nterrupt_mask is also used to inhibit user access to supervisor 
messages. If the nth bit of the n±_rec_interrupt_mask register is 1, them if 
a Data Network message with a tag value of n arrives (say via the LDR interface) 
the message is effectively invisible and inaccessible to user programs. Specifi¬ 
cally, the rec_ok flag will be 0 when read by the user, and an attempt by the user 
to read from the receive FIFO will fail, as though the FIFO were empty. When 
the supervisor attempts to read the message, however, the rec_ok flag will have 
the correct value, and reading from the receive FIFO will receive the message as 
usual. 
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Using CMost Commands to Set Up NI Interrupt Handlers 

You can use CMOST commands to instruct the NI to signal an interrupt when it 
receives a message with a specific tag. This interrupt causes the processing node 
to execute a specific routine of your program. The CMOS_signal system call is 
used to set up an interrupt: 

CMOS_signal( signal, user Junction, tagjnask ) 

The signal argument is the signal type, and must be the predefined constant 
SIGMSG. The user Junction argument is the name of a user-defined function that 
should handle receiving and processing the message. The tagjnask argument is 
a 16-bit field, one bit for each possible value of the tag. If bit n in this mask is 
set, then the receipt of a message with a tag of n causes user Junction to be ex¬ 
ecuted. (Remember that you are limited to using only the first four bits of this 
mask, corresponding to the tags 0 through 7.) 

So, for example, the function call 

CMOS_signal( SIGMSG , my_msg_handler , OxFE); 

arranges the NI interrupt system so that when a Data Network message with a tag 
from 1 to 7 is received, the user-defined procedure my_msg_handler is called. 

Note: To use the CMOS_signal function, you must #include the file cm/ 
cm_s±gnal .h. For more information on CMOS_signal, see the UNIX manual 
page for the function. (This is included as Appendix E to this document.) 


Tag Fields and the Message-Counting Registers 

Tag fields also allow system software to automatically maint ain a count of mes¬ 
sages sent and received by the NI. This is a key part of the network-done feature 
of the Control Network (see Section 4.2.9). It allows the NI to determine quickly 
when the Data Network is clear of user messages. Two registers are used to con¬ 
trol this message-counting feature: 

ni_dr_me s a age_c ount Register, contains current message count. 
ni_ccnmt_mask Register, contains tag-count enable flags. 
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Message Count Disabling 

The ni_dr_message_count register contains a signed 32-bit integer value that 
is incremented when a Data Network message is sent (by any of the three inter¬ 
faces), and decremented when a message is received. 

When the message_coxmt register becomes zero for all non-abstaining nodes, 
the NI assumes that there are no countable messages in transit in the Data Net¬ 
work. It is possible to disable message counting for messages with specific tag 
values. (This is useful, for example, if you only wish to keep a count of user 
messages, and want supervisor messages to go uncounted.) 

The ni_count_mask register controls this enabling and disabling of message 
counting. It contains 16 flags, one for each tag value. If the nth count_mask bit 
is 1, then messages with a tag of n are counted by ni_dr_message_count. If 
the nth bit is zero, messages with that tag are not counted. 

It’s important to be sure that the sending and receiving nodes for a message agree 
on whether the message’s tag should or should not be counted; if they do not 
agree, the ni_dr_message_count register’s value is useless, and can wrap 
around, becoming negative — see the discussion of this situation below. 

Note: The supervisor can write a value to ni_drjnessage_count, for exam¬ 
ple, to set the register back to zero, but this should only be done when the Data 
Network is not in use. Otherwise, there is no way to guarantee that the value of 
this register remains the same as the value that was written into it. 


Negative Message Count interrupts 

If the sum of the message_count registers for all nodes becomes negative, it 
means that either a message was lost in transit or was counted incorrectly. If the 
global message_count sum is negative when a Data Network operation is 
attempted, a Yellow interrupt (dr count negative) is signaled. (See Section 
B.3.4 in Appendix B.) 

Note: If the message_count register is incremented or decremented beyond its 
32-bit signed value capacity, its value “wraps around,” becoming negative. How¬ 
ever, the register is large enough that this should not happen unless there is a 
serious error (a hardware problem that causes messages to be lost, nodes that do 
not agree on counting of tag messages, etc.). 
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3.5.5 The Send and Receive State Fields 

The DR interface is mutually exclusive with the LDR and RDR interfaces. It is 
an error to try to write a message to the DR send FIFO while there is a partially 
completed message in either the LDR or RDR send FIFOs. Likewise, having a 
partially completed message in the DR send FIFO makes it an error to try to send 
a message via the LDR or RDR FIFOs. In either case, the status registers and 
FIFOs of the excluded interface(s) are invalidated. 

You can use the ni__dr_send_state field to determine which interfaces are in 
use. The value of this field is an integer from 0 to 2, with the following me aning s: 

0 No partial messages in any send FIFO. 

1 Partial message in the DR send FIFO. 

2 Partial message in either or both of the LDR or RDR send FIFOs. 

There is also a corresponding ni_dr_r ec_s tate field that you can use to deter¬ 
mine which receive interfaces are in use. (However, because the DR interface 
cannot be used to receive messages, this field is not as useful as 
ni_dr_send_state.) The value of the ni_dr_rec_state field is again an 
integer from 0 to 2: 

0 No partial messages in any receive FIFO. 

1 Reserved. (The DR interface cannot receive messages.) 

2 Partial message in either or both of the LDR or RDR receive FIFOs. 

You can obtain the values of these fields by using the following macros: 

DR_SEND_STATE { Status) 

DR_RECEIVE_STATE ( status ) 

For example, 

• int value = CMNA_LDR_status(); 
int send_state = DR_SEND_STATE(value); 
int rec_state = DR_RECEIVE_STATE(value); 

Implementation Note: The n±_dr_send_state and ni_dr_rec_state 
fields exist only for the DR interface (that is, are accessible only from the 
ni_dr_status register). 

Note: The two half-network interfaces are not mutually exclusive. There is no 
restriction on having partially completed messages simultaneously in the LDR 
and RDR FIFOs. (This kind of simultaneous message sending is one reason that 
the LDR and RDR interfaces exist.) 
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3.5.6 The Network-Done Flag 

The ni_router_done_costplete flag is used by the Control Network as part 
of its network-done message function. This feature is designed to make it easy 
to synchronize the nodes after a Data Network operation. 

You can use the following macro to access this flag: 

DR_ROUTER_DONE ( Status) 

For example, 

int value = CMNA_LDR_status(); 

int network_done - DR_ROUTER_DONE(value); 

As noted above, the message-counting register ni_dx_me s s age__c ount also 
plays a part in the network-done feature. For more information on network-done 
messages, see Section 4.2.9. 


3.6 The Private Register 


The private register for each of the network interfaces contains the following 
subfields: 


n±_dinteifacejpx i va t e 
ni_rec_ok_ie 
niJLock 
ni_rec_stop 
ni_rec_£ull 
ni_dr_rec_al l_£al l_down 
ni_all_£all__down_ie 
ni_all_f all_down_enaJble 
ni_sf ifo_goeB_empty_ie 
ni_rdone_cosrplete_ie 


Private register. 

Flag, “Receive OK” interrupt enable, 
Interface lock flag. 

Interface stop flag. 

Flag, indicates receive FIFO is full. 
Flag, set for All Fall Down message. 
All Fall Down interrupt enable flag. 
Flag, triggers All Fall Down mode. 
Send FIFO empty interrupt enable. 
Network-done interrupt enable. 


The rec_ok_ie, lock, rec_stop, and recjEull subfields are as described 
in Chapter 2. The remaining three fields are used to control the All Fall Down 
mode feature of the Data Network, as described in Section 3.7 below. 


Note: The subfield ni_rec_stop is accessible only from the DR interface (that 
is, its value is defined only for the nl_dr_private register). 
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3.7 All Fall Down Mode 

All Fall Down mode is a feature of the Data Network that is used primarily by 
the supervisor for swapping processes out of partitions. When All Fall Down 
mode is triggered within a partition of the Data Network, all messages currently 
in transit within that partition are immediately routed downwards through the 
network to the nearest possible node, regardless of their actual destination. This 
process clears the Data Network of pending messages as swiftly as possible. 

The three private register subfields, ni_dr_rec_all_fall_down, 
ni_all_fall_down_ie, and ni_all_f all_down_enable, are used to trig¬ 
ger All Fall Down mode, as well as to detect when an arriving Data Network 
message is the result of All Fall Down mode. 


3.7.1 Triggering Ail Fall Down Mode 

, To trigger All Fall Down mode in a partition, each node in the partition should 
set its ni_all_f all_down_enable flag to 1. This informs the Data Network 
hardware that the NIs are ready to receive All Fall Down messages. 

For the Curious: The Data Network is organized in layers, with each layer man¬ 
aged by internal switching nodes. When All Fall Down mode is started by the 
nodes, it is broadcast through all the layers of the Data Network, causing the 
internal switching nodes to begin routing messages downward and out of the net¬ 
work. The Data Network is designed in a fault-tolerant manner, so that even if 
a given Data Network switching node is not yet in All Fall Down mode, an All 
Fall Down message sent through it by a higher level node “falls through” and 
continues moving toward the processing nodes. 


3.7.2 Detecting Ail Fall Down Mode Messages 

The flag ni_dr_rec_all_fall_down is set whenever the current message in 
the receive FIFO is the result of an All Fall Down operation. 

You can also have the NI trigger an interrupt when an All Fall Down message 
becomes available in the receive FIFO (either by arriving at an empty FIFO, or 
by being brought forward after a preceding message has been read out). If the 
interrupt enable flag n±_all_f all_down_ie is set, the arrival of an All Fall 
Down message triggers a Green interrupt (dr rec all fall down). 
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3.7.3 Resending All Fall Down Mode Messages 

Each message re-routed by Ail Fall Down mode carries with it enough informa¬ 
tion so that the receiving node can resend the message to its intended destination. 
When an All Fall Down message is read from the receive FIFO, the first word 
read is not the first word of the message itself, but is an extra address word, con¬ 
taining information about the intended destination of the message. 

The All Fall Down address word has the following format: 

n _0 

offset ' 

where 

header is a 4-bit header giving the length of the offset field 
tag is the original tag field of the message 
length is the message length in words, excluding the address word 
offset is an n-bit field used to construct the real address 

The header field indicates the length of the offset field, but in a slightly convo¬ 
luted manner. The length of the offset field, n, is 4 times the least integer not less 
than one-half of the header value, h. In equation form: 



(An algorithmic way to get ths result is to take bits 29 - 31 of the header field 
as an integer, arithmetically add bit 28, and left-shift the result by two bits.) 

Once you have the offset length, take the physical address of the current node and 
replace the least significant n bits with the n-bit value from the offset field. This 
gives the destination physical address. For example, if the header value is 1, then 
the offset is 4 bits in length. If the offset value is OxC, and the physical address 
of the current node is 0x00111, then the destination physical address is 0x0011C. 

The tag and length fields duplicate the values obtainable from the rec_tag and 
rec_length fields in the status register. However, these fields are included 
in the All Fall Down address word because programmers may find them useful. 

Note: When an All Fall Down message is received, the value of the 
rec_length field is equal to the original length of the message — the number 
of data words in the FIFO not counting the All Fall Down address word. How¬ 
ever, the rec_length_lef t field contains the total number of words left in the 
receive FIFO, and this count includes the All Fall Down address word. 
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3.8 Interrupt Enable Flags 

There are two interrupt enable flags in the n±_dinterface_px ivate register. 

The ni_sf if o_goes_empty_ie flag controls whether a Green interrupt (send 
FIFO empty) is signaled when any Data Network send FIFO goes empty. The 
ni_rdone_complete_ie flag controls whether an Orange interrupt (router 
done complete) is signaled when a network-done operation completes. 


3.9 Data Network Usage Note: Receive before You Send 

An important strategy to keep in mind when using the Data Network is “Receive 
before you send.” That is, in most cases you should structure your code so that: 

■ Each node attempts to read a message from the Data Network before send¬ 
ing a new message into it. 

■ If a node is unable to send a message, the node attempts to read a message 
to help decrease the network load. 

The Data Network has a large capacity for messages from nodes, but the sheer 
number of nodes connected to it can overwhelm it if the nodes send messages 
into the network without attempting to receive them. Your code should be biased 
toward removing messages from the network rather than adding them. Your code 
should also provide fair opportunities for both receiving and sending, where 
“fair” means the ratio between the two should be bounded both below and above, 
and where “opportunity” means the opportunity to attempt sending or receiving 
a message, whether or not the attempt is successful. Thus, the sending and re¬ 
ceiving portions of your code should be called with fairly equal frequency. 

When you are using the LDR and RDR concurrently, you should likewise main¬ 
tain a balance in using both interfaces, so that neither interface becomes more 
heavily loaded than the other. In short, the rule of thumb is: “Receive before you 
send, but receive and send fairly.” 

Note: Some applications use the LDR and RDR interfaces for completely differ¬ 
ent purposes, and thus do not normally maintain a load balance between the two 
halves of the Data Network (that is, one network interface may be used less often 
than the other). Nevertheless, such application code should still try to maintain 
a receive/send balance within each of the two network interfaces. 
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3.10 Examples 

The examples shown below are code fragments intended to be run on the pro¬ 
cessing nodes. See Chapter 7 for a discussion of large-scale program structure. 

Also, since the interfaces for the DR, LDR, and RDR are virtually identical, the 
examples below are written for the LDR only. Appropriate functions for the other 
network interfaces can be obtained by appropriate substitution of names. 


Sending and Receiving a Message 


Here is a pair of functions that send and receive messages via the LDR interface. 
The message is assumed to be composed of length words of data, and is sent with 
the specified tag value to the node with the given dest_address. 


int LDR_send (dest_address, message, length, tag) 
unsigned dest_address, tag; 
int *message; 
int length; 

{ int i; 


CMNA_ldr_send_first(tag, length, dest_address); 
while (length—) CMNA_ldr_send_word(* (message+ + ) ); 
return (SEND_OK(CMNA_ldr_status())); 

1 


/* Highest tag NOT currently assigned as interrupt */ 
int tag__limit = 0; 

int LDR_receive (message, length) 
int *message; 
int length; 

{ 

int i, tag = 999; 

/* Skip messages currently assigned as interrupts 

*/ 

while (tag>tag_limit) { 

if (RECEIVE_OK(CMNA_ldr_status())) 

tag = RECEIVE_TAG(CMNA_ldr_status()); 

> 

while (length—) 

*(message++) = CMNA_ldr_receive_word(); 
return (tag); 
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For example, the following code fragment causes each node to send a message 
to the node with the next-higher node address. (The node with the highest ad¬ 
dress sends a message to node 0.) 

int next_node = (CMNA_self_address + 1) 

% CMNA_partition_size; 
int i, message[MAX_ROUTER_MSG_WORDS]; 
for (i = 0, i<MAX_ROUTER_MSG_WORDS, i++) message[i]=i; 
LDR_send(next__node, message, MAX_ROUTER_MSG_WORDS, 0); 
LDR receive( message, MAX_ROUTER_MSG_WORDS ); 


Sending and Receiving Long Messages 

Of course, the functions above are limited by the size restriction on Data Net¬ 
work messages. If you have a lot of data to send, you’ll probably want to use a 
function that can send a message of any word length, breaking it up into chunks 
as appropriate. Here’s such a function, which handles both sending and receiving 
the message in a single function call: 

/* Send/Receive function with no length restriction 
*/ 

LDR_send_receive_msg(dest_address, message, length, 

tag, dest) 

unsigned dest_address, tag; 
int *message, *dest; 
int length; 

{ 

int packet_s ize=MAX_ROUTER_MSG_WORDS-1; 
int send_size, receive_size; 
int offset, source_offset=0, dest_offset; 
int words_to_send=length, words_received=0; 
int count, rec_tag, status; 

while ((words_received<length)||(words_to_send)) 

{ 


/* First try to receive a packet */ 
status=CMNA_ldr_status() ; 
if (words_received<length && 
RECEIVE_OK{status) && 

RECEIVE_TAG(status) <= tag_limit) { 
dest offset = CMNA ldr receive word(); 
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receive_size= 

RECEIVE_LENGTH_LEFT(CMNA_ldx_StatUS()); 
for (count=0; count<receive_size; count++) 

dest[dest_offset++]=CMNA_ldr_receive_word(); 
words_received += receive_size; 

} 

/* Now try sending a packet */ 
if (words_to_send) { 

send_size = ((words_to_send < packet_size) ? 

woxds_to_send : packet_size); 

do { 

CMNA_ldr_send_first(tag, send_size+l, 

dest_address); 

/* Send offset of msg data being sent */ 
CMNA_ldr_send_woxd(source_offset); 
offset=source_offset; 

for (count=0; count<send_size; count++) 
CIfflTA_ldr_send_word(message [offset++]) ; 

} while (!SEND_OK(CMNA_ldr_status())); 
source_offset-offset; 
words_to_send -= send_size; 

} /* if */ 

} /* while */ 

} 

Here’s an example of how to call this function: 

#define LONG_FACTOR 5 

int mirror_node = (CMNA_partition_size-l) - 

CMNA_s e1 f _addr ess; 

int i, length = MAX_ROUTER_MSG_WORDS*LONG_FACTOR; 
int send[MAX_ROUTER_MSG_WORDS *LONG_FACTOR]; 
int receive[MAX_ROUTER_MSG_WORDS*LONG_FACTOR]; 

for (i=0, i<length, i++) long_message[i]=i; 

LDR_send_receive_msg(mixror_node, send, 

length, 0, receive); 
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Interrupt-Driven Message Retrieval 

Using interrupt-driven message retrieval simply requires that you define a han¬ 
dler to be called when an interrupting message arrives. The handler should take 
no arguments, and its returned value is ignored. 

/* Message handler for interrupt-driven LDR test */ 

#include <cm/cm_signal.h> 

int interrupt_done = 0; 

int interrupt_expect_length; 

int interrupt_receive[MAX_ROUTER_MSG_WORDS]; 

void LDR_receive_handler () 

{ 

int temp=tag_limit; 
tag_limit=3; 

LDR_receive{interrupt_receive, 

interrupt_expect_length); 
tag_limit=temp; 
interrupt_done=l; 

} 

You use CMOS_signal to inform the NI that it should signal an interrupt from 
some or all of the possible tag values. (Remember that you must #include the 
header file cmsys/cm_signal to have access to CMOS_s±gnal.) For example: 

int i, next_node, message_length=MAX_ROUT- 
ER_MSG_WORDS; 

int message[MAX_ROUTER_MSG_WORDS]; 

for (i=0, i<message_length, i++) message[i]=i; 

next_node = (CMNA_self_address+l) 

% CMNA_partition_s iz e; 

/* signal interrupts for non-zero tag values */ 

CMOS_signal( SIGMSG , LDR_rece±ve_handler , 14 ); 

/* Send message with an interrupt tag (3) */ 
interrupt_done = 0; 

interrupt_expect_length = message_length; 

LDR_send(next_node, message, message_length, 3); 

/* Wait for handler to signal interrupt finished */ 
while (interrupt_done==0) {}; 
printf("Received message: "); 
for (i=0, i<message_length, i++) 
printf("%d ", message[i]); 
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Sending via LDR and RDR Simultaneously 

One advantage to having the two sub-interfaces in the Data Network is that you 
can send messages simultaneously through the LDR and RDR. For example, 
here’s a pair of functions that send a single message via both interfaces, compar¬ 
ing the received results to make sure that the message was received properly: 

/* Send/Receive functions using LDR/RDR in tandem */ 
void LDR_RDR_send (dest_address, message, length, 
tag) 

unsigned dest_address, tag; 
int *message, length; 

{ 

int i; 

CMNA_ldr_send_first(tag, length, dest_address); 
CMNA_rdr_send_first(tag, length, dest_address); 
for (i=0; i<length; i++) { 

CMNA_1dr_s end_wor d(message[i]); 

CMNA_rdr_send_word(message[i]); 

} 

} 

int LDR_RDR_receive (message, length) 
int *message, length; 

{ 

int i, ldr_value, xdr_value, length_received_ok=0; 
while ( ! RECEIVE_OK (CMNA__ldr_status () ) | | 

!RECEIVE_OK(CMNA__r dr_s tatus())) {} 

for (i=0; i<length; i++) { 

ldr_value=CMNA_ldr_receive_word(); 
rdr_value=CMNA_rdr_receive_word () ; 
if (ldr_value==rdr_value) { 
message[i]=ldr_value; 
length_received_ok++; 

} 

} 

return(length_received_ok); 

} 
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The Control Network consists of three interfaces, the broadcast interface (BC), 
the combine interface (COM), and die global interface. 



Figure 13. The three interfaces of the Control Network: BC, COM, and global. 


The broadcast and combine interfaces are very similar, and there are some inter¬ 
nal interactions between these two interfaces that you’ll need to keep in mind. 
The global interface, however, is different in both structure and purpose from 
either of the other two interfaces. 

This chapter describes the three Control Network interfaces, and presents the 
registers that are used to manipulate them. 
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4.1 The Broadcast Interface 

The broadcast interface is used to broadcast a message from a single source node 
to all nodes in the same partition (including the broadcasting node). 

The broadcast interface provides two separate register interfaces, one for user 
broadcasts (BC), and one for supervisor broadcasts (SBC). The two register in¬ 
terfaces are completely independent, and can be used concurrently to broadcast 
messages. Where the sections below refer to “broadcast messages” generically, 
the description applies equally and independently to both the user and supervisor 
interfaces. 

Implementation Note: Because of the way the broadcast and combine interfaces 
interact, if a node is abstaining from a combine operation, that node should not 
execute a broadcast operation until the combine operation is completed. (For 
more information, see Section 8.2.6.) 


4.1.1 Broadcast Register Interfaces 

The two broadcast register interfaces are based on the generic model presented 
in Chapter 2. The only difference between them is that the supervisor broadcast 
registers can be accessed only from the supervisor area. 

The following NI registers form the broadcast interface: 

ni_binterface_send_£. ir s t Used to send the first value of a message. 

ni_binterface_aezid Used to send the rest of the message. 

xi±_binterface_xecv Used to receive a message. 

ni_binterface_stat\iB Status register. 

ni_binterface_contxol Control register. 

n±_binterface_px iva.te Supervisor control register. 

The binterface part of these names is a unique abbreviation for each interface: 

be - user broadcast interface sbe - supervisor broadcast interface 

The purpose and use of each of these registers is described in the sections below. 
Figure 14 contains a memory map showing the relative locations of these regis¬ 
ters in the user and supervisor areas. 
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The Broadcast Interface Registers at a Glance: 


hex offsets 
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Figure 14. NI registers associated with each of the broadcast interfaces. 


4.1.2 Broadcast Messages 

A broadcast message is essentially synchronous — a single node broadcasts a 
message that is received by all nodes in its partition (including the broadcasting 
node itself). 

Only one node in each partition can broadcast via a given interface at any time. 
If two or more nodes in the same partition attempt to broadcast simultaneously, 
via the same interface (user or supervisor), the effect is unpredictable. An error 
may be signaled and/or transmitted data may be lost. (Remember, however, that 
the user and supervisor broadcast interfaces operate independently, and can be 
used concurrently by different nodes in the same partition.) 
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Broadcast messages are atomic with respect to sending; a broadcast message is 
not transmitted until all its component words have been written to the send FIFO. 
Broadcast messages are not atomic in transit, however. A multiword message 
may be split in transit into two or more smaller messages. In addition, as broad¬ 
cast messages arrive at each node they are concatenated together in the receive 
FIFO. 

From the point of view of each receiving node, it always appears as if there is 
exacdy one broadcast “message” waiting to be read from the receive FIFO. Once 
a node begins receiving a message (that is, when it examines the status register 
to determine the length of message that is available), the length of the message 
is fixed, and a new “message” is formed behind it in the FIFO from any words 
that arrive while the first message is being read out. 

Although the length of a broadcast message is not maintained, the order of the 
words within a message is maintained, as well as the order of messages sent and 
received via the same interface, user or supervisor. (There is no predictible rela¬ 
tionship, however, between the deliveries of user and supervisor messages to the 
same node. Effectively, the two interfaces act as independent “streams” of mes¬ 
sages.) 

Usage Note: The broadcast interface is designed in such a way that a message 
is not removed from the send FIFO before all non-abstaining nodes have received 
it. This feature can be used to force synchronization of the nodes. 

Implementation Note: Each broadcast interface’s private register includes a 
supervisor flag, ni_send_enable, which controls whether broadcast sending 
is enabled via that interface. (See Section 4.1.8 for a description of these flags.) 


4.1.3 Sending Broadcast Messages 

A broadcast message consists of a series of one or more words. The maximum 
length allowed for a message is determined by the length limit of the send FIFOs. 
The only auxiliary information associated with a broadcast message is its length. 
However, the length is only meaningful for the node that sends a message, be¬ 
cause of the way messages can be split and concatenated in transit. 

Programming Note: The length limit of the broadcast send FIFOs is given by 
the constants max_broadcast_msg_words and max_sbc_msg_words (cur¬ 
rently 4 for both interfaces). 
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The following FIFO registers are used to send messages: 

n±_binterface_aend_£ixat Used to send the first value of a message. 

ni_binterface_aend Used to send the rest of the message. 

and there are corresponding send_f irst and send macros: 

CMNA_bc_send_f irst ( length, value) 

CMNA_bc_send_f irst_double ( length, value) 

CMNA_bc_send_word {value) 

CMNA_bc_send_£loat [value) 

CMNA_bc_send_doub 1 e (value) 

For the send_f irst macros, the length argument is the length of the message 
in words, and value is the first value of the message. For the send macros, value 
is the second and succeeding values of the message. 


4.1.4 Auxiliary Information 

The auxiliary data field of a broadcast message (BC or SBC) has the form 


8 0 



where length is the length of the message in words. The length field can have any 
value from 1 up to max_broadcast_msg_words or max_sbc_msg_words . 
(The high-order bits of the auxiliary data have no useful me anin g, but must al¬ 
ways be specified as 0.) 

The following constant specifies the starting bit position of the length field: 
NI_BC_SEND_AUXILIARY_LENGT H _P The length field offset (0). 


4.1.5 Receiving Broadcast Messages 

Broadcast messages are received as described in Chapter 2. For each broadcast 
interface, the following register is used to receive messages: 

n±_binterface_xecv FIFO register from which values are read. 
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To receive a message from the broadcast interface, use the network-specific read¬ 
ing operations described in Chapter 2: 

value = CMNA_bc_receive_wor d. () ; 
value = CMNA_bc_receive_float(); 
value = CMNA be receive double(); 


4.1.6 The Broadcast Status Register 

The status registers for each of the interfaces contain the following subfields: 

ni_binterface_st&tus Status register. 

ni_send_ok Flag, status of message being sent. 

nl__send_space Field, space left in said FIFO. 

ni_send_empty Flag, indicates empty send FIFO. 

ni_rec_ok Flag, indicates receipt of message. 

ni__rec_length_lef t Field, words left in the FIFO. 

The meanings of these sub-fields are as described in Chapter 2. You can obtain 
the values of these sub-fields by using the generic field extractors described in 
Chapter 2 (Section 2.5.4). 

The macro used to get the value of the broadcast status register is 
int value = CMNA_bc_status() 

Note: As described in Section 1.4 in the Appendixes, the bit length of the 
length_le£t field has changed; to access this field, you should use the macro 
bc_receive_length ( status-value) for both the BC and SBC interfaces. 


How to Interpret the Value of the “Length Left” Field 

The NI combines broadcast messages as they are received, so there is never more 
than one “message” waiting to be read from the receive FIFO. However, broad¬ 
cast messages are never appended to a message that is in the process of being 
retrieved, so you needn’t worry that a message will grow unexpectedly. 

Once you have retrieved the first value of a received message, it is safe to assume 
that reading a number of words equal to the rec_length_le£t value retrieves 
the rest of the message. (Remember, however, that this method is not guaranteed 
to read all words of a multiword message that was divided in tr ansi t.) 
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4.1.7 Abstaining from the Broadcast interface 

Each broadcast interface has an abstain flag that you can use to cause the NI to 
ignore incoming broadcast messages. The abstain flag’s effects and use are as 
described in Section 2.6. 

n±_binterface_contxal Status register, contains rec_abstain field. 
ni_rec_abstain Flag, broadcast interface abstain flag. 

The address constant for the abstain register is bc_control_reg. You can use 
the macros described in Section 2.6.3 to read and write the abstain flag: 

value = CMNA_read_abstain_f lag (bc_control_r eg) ; 

CMNA_write_abstain_flag(be control_reg, value) ; 


4.1.8 The Broadcast Private Register 


The private register for each broadcast interface contains the following subfields: 


n±_binterface_£x ivat e 
ni_rec_ok_ie 
ni_lock 
ni_send_stop 
n±_rec__£ull 
ni send enable 


Private register. 

Flag, “Receive OK” interrupt enable. 
Interface lock flag. 

Interface stop flag. 

Flag, indicates receive FIFO is full. 
Flag, enables/disables send FIFO. 


The rec_ok_le, lock, send_stop, and rec_£ull subfields are as described 
in Chapter 2. The remaining field is described below. 


The Send Enable Flag 

Each broadcast interface has an ni_send_enable flag, which is used to enable 
and disable the broadcast send FIFO. When this flag is set to 1, message sending 
is permitted. When the flag is set to 0, an attempt to write a message to the send 
FIFO signals a Bus Error. The send_enable flag should be changed only when 
there are no broadcast messages pending for the interface. 

Usage Note: While this flag can be used as a kind of “send abstain” flag to 
ensure that only one node broadcasts at any given time (that is, by disabling send¬ 
ing for all nodes but the one making the broadcast), it is much simpler to 
structure your code so that only one node is permitted to broadcast at any time. 
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Implementation Note: The CMost operating system sets the send_enable 
flag for the broadcast interface (but not the supervisor interface) to 0 by default. 
This flag must be set to 1 to permit broadcasting of messages. To turn on this 
flag, you can use the following C macro call; this call must be made prior to any 
broadcast interface operations: 

CMNA_participate_in (NI_BC_SEND_ENABLE) ; 


4.1.9 Broadcast Interface Examples 

The examples shown here are fragments of code intended to be run on the pro¬ 
cessing nodes. See Chapter 7 for a discussion of large-scale program structure. 


Sending and Receiving a Message 

These functions send and receive messages via the broadcast interface. The mes¬ 
sage is assumed to be composed of length words of data starting at the memory 
location specified by message. 

int BC_send(message, length) 
int *message, length; 

{ 

int i; 

CMNA_bc_send_first(length—, *message++); 
for (i=0; i<length; i++) 

CMNA_bc_send_word(*message++); 
return(SEND_OK(CMNA_bc_status())); 

1 

int BC_receive(message, length) 
int *message, length; 

{ 

int i; 

for(i = 0; idength; i + +) { 

while(!RECEIVE_OK(CMNA_bc_status())) {} 

message[i] = CMNA_bc_receive_word(); } 

return(length); 

} 
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For example: 

int i, message[MAX_BROADCAST_MSG_WORDS]; 
for (i=0, i<MAX_BROADCAST_MSG_WORDS, i++) 
message[i]=i; 

BC_send(message, MAX_BROADCAST_MSG_WORDS); 
BC_receive(message, MAX_BROADCAST_MSG_WORDS); 


4.2 The Combine interface 

The combine interface is used for executing operations that combine in parallel 
a single value from each processing node. 

These are the supported operations: 

■ parallel prefix (scanning), which performs a cumulative operation (addi¬ 
tion, maximum, logical AND, etc.) over the values from each node in 
either increasing or decreasing order of send addresses 

■ reduction, which combines the values from all the nodes and then returns 
this single combined result to all participating nodes 

■ network-done, which simplifies the task of synchronizing the nodes after 
a Data Network operation 

Each operation is described in more detail below. 

Implementation Note: Because of way the broadcast and combine interfaces 
interact, if a node is abstaining from a combine operation, that node should not 
execute a broadcast operation until the combine operation is completed. (For 
more information, see Section 8.2.6.) 


NI Version 2.2 (CM-5E), June 1994 

Copyright © 1994 Thinking Machines Corporation 


71 



NI Programmer’s Handbook 


4.2.1 The Combine Register Interface 


The combine interface’s register interface is based on the generic model pres¬ 
ented in Chapter 2, and includes the following registers: 


n±_com_send_fir s t 
ni_com_send 
ni_com_recv 
n.i__com_s t a tus 
ni_com_control 
nl co m p rivate 
ni scan start 


Used to send the first value of a message. 

Used to send the rest of the message. 

Used to receive a message. 

Status register. 

Control register. 

Supervisor control register. 

Control register used to set scanning segments. 


The purpose and use of each of these registers is described in the sections below. 
Figure 15 contains a memory map showing the relative locations of these regis¬ 
ters in the user and supervisor areas. 


The Combine Interface Registers at a Glance: 


hex offset 


ni com send first 




ni com send 


ni com rec 


ni com control 


nicomprivate 


ni com status 


0x5000 

0x0A4 0 
0x0A3 0 
0x0A20 

OxOAlO 

oxoaos 

OxOAOO 


ni_interface_send_first Addressing Pattern 


user/supervisor bit 
NI base address 


interface 

index 


COM 


X 

0000 

1 

0 

H 

{pattern| combiner j 

length 

000 


31 20 

' 19 1 

18 15 

‘ 14 


12 

11 10 ! 9 7 ' 6 

3 

2 0* 


Figure 15. NI registers associated with the combine interface. 
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4.2.2 Combine Messages 

The combine interface is essentially synchronous — combine operations are not 
completed until all non-abstaining nodes have sent the same type of combine 
operation. If two nodes attempt to start different combining operations at the 
same time, a Yellow interrupt (be or com collision) is signaled. Once this 
interrupt has been signaled, combine messages are no longer guaranteed to be 
valid — it is necessary to flush the Control Network to restore normal operation 
(see the discussion of Control Network flushing in Section 6.4). 

Combine messages are atomic in both sending and receiving; a combine message 
is not transmitted until all its component words have been written to the send 
FIFO, and arrival of each message is not reported until all the words of the mes¬ 
sage have arrived in the receive FIFO. 

The order of combine messages is strictly preserved in transit. With the exception 
of the network-done operation, which uses a different mechanism, the results of 
combine operations are delivered into the receive FIFO in the same order the 
operations were started. 

Combine operations can be pipelined. Although all nodes must start the same 
combine operation in order for that operation to complete, nodes are not required 
to read the results of each combine message before sending the next. The length 
of the pipeline is limited only by the capacity of the message FIFOs. 

Important: Pipelined messages cannot use doubleword read/wiite operations — 
see Section 8.1.2. 


4.2.3 Sending Combine Messages 

A combine message consists of a series of one or more words, with the exception 
of network-done messages, which are always 1 word in length. The maximum 
length allowed for a message is determined by the length limit of the send FIFO. 

Progr ammin g Note: The length limit of the combine interface send FIFO is giv¬ 
en by the constant max_combine_msg_words (currently 5). 

The following FIFO registers are used to send messages: 

ni_com_send_f irst Used to send the first value of a message. 
ni_com_send Used to send the rest of the message. 
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and there are corresponding send_£irst and send macros 

C34NA_com_send_first (combiner, pattern, length, value) 
CMNA_com_send_f ir st_double ( combiner, pattern, length, value) 

CMNA_com_send_word (value) 

CMNA_com_send_f loat (value) 

CMNA_com_send_double (value) 

For the send_f irst macros, the length argument is the length of the message 
in words, and value is the first value of the message. The combiner and pattern 
arguments are described in the sections below, covering each of the possible 
combine operations. 

For the send macros, value is the second and succeeding values of the message. 


4.2.4 Auxiliary information 


The auxiliary information has three components: the length of the message in 
words, a three-bit combiner value, and a two-bit pattern value. (The legal com¬ 
biner and pattern values are described below.) 


The auxiliary data field of the message has the form 


where 


8 


0 


pattern 


combiner 


length 


pattern is a two-bit value selecting the order in which values are combined 
combiner is a three-bit value selecting the combine operation performed 
length is the length of the message in words 

The following constants specify the starting bit positions of these fields: 


NI_COM_S END_AUXILIARY_PATTERN_P 

NI_COM_SEND_AUXILI ARY_COMB INER_P 

NI COM SEND AUXILIARY LENGTH P 


The pattern field offset (7). 
The combiner field offset (4). 
The length field offset (0). 
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To construct a send_first address, add the following values: 

The pattern value: pattern « NI_com_send_auxil i ary_patternjp 

The combiner value: combiner « ni_com_send_auxiliary_combiner_p 
The length value: length « NI_com_send_auxiLI ARY_LENGTH_P 


4.2.5 Legal Combiner and Pattern Values 

For scans and reductions, these are the legal pattern and combiner values: 


pattern : 

1 — Backward scan (combine in decending order of node address). 

2 — Forward scan (combine in increasing order of node address). 

3 — Reduction operations. 

combiner. 

0 — Bitwise inclusive OR. 

1 — Signed addition. 

2 — Bitwise exclusive OR. 

3 — Unsigned addition. 

4 — Signed maximum. 

A pattern value of 0, together with a combiner value of 5, specifies a network- 
done operation, described later in this chapter. The combiner values 6 and 7 are 
not currently used. 

The following constants can be used to specify the value of the pattern field: 

s can_forward Forward scan pattern (2). 

s CAN_BACKWARD Backward scan pattern (1). 

scan_reduce Reduction scan pattern (3). 

SCAN_ROUTER_DONE Network-done operation (0). 

The following constants can be used to specify the value of the combiner field: 


OR_SCAN 

ADD_SCAN 

XORJ3CAN 

UADDJ3CAN 

MAX_SCAN 

ASSERT ROUTER DONE 


Inclusive OR (0). 

Signed addition (1). 
Exclusive OR (2). 

Unsigned add (3). 

Signed maximum (4). 
Network-done operation (5). 


The length field can have any value from 1 up to max_combine_msg_words. 
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4.2.6 Receiving Combine Message 

The message-receiving interface of the combine interface is as described in 
Chapter 2, with the exception of the network-done operation, which is received 
through the Data Network status field n±_router_done_complete (see Sec¬ 
tion 4.2.9). 

The following register is used to receive combine messages: 

ni_com_recv FIFO register from which values are read. 

To receive a message from the combine network, use the network-specific read¬ 
ing operations described in Chapter 2: 

value = CMNA_com_receive_word(); 
value = CMNA_com__r ece ive_f 1 oat () ; 
value = CMNA com rece±ve_double(); 


4.2.7 The Combine Status Register 


The combine status register contains the following subfields: 


nl_com_s tatus 
n±_send__ok 
nl_send_space 
n±_send_empty 
ni_rec_ok 
ni_rec_length 
ni_rec_length_lef t 
n± com scan overflow 


Status register. 

Flag, status of message being sent. 
Field, space left in send FIFO. 
Flag, indicates empty send FIFO. 
Flag, indicates receipt of message. 
Field, length of message in words. 
Field, words left in the FIFO. 

Flag, indicates add-scan overflow. 


The send_ok, send_space, send_empty, rec_ok, rec_lengtli, and 
rec_length_lef t subfields are as described in Chapter 2, and you can obtain 
the values of these sub-fields by using the generic field extractors described in 
that chapter. The remaining flag, com_scan_over£low, is described in Section 
4.2.8. 


Use this C macro to get the value of the combine status register: 
int value = CMNA com_status() 
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4.2.8 Scanning (Parallel Prefix) and Reduction Operations 

In a scan or reduction operation, each node sends a single value that is combined 
with the values sent by the other nodes in the partition. A scan or reduction mes¬ 
sage is from 1 to 5 words in length, representing a value to be combined. 

When each participating node has sent a value, the values are combined accord¬ 
ing to the combiner and pattern in the auxiliary data of the message, and the 
result is delivered after a brief interval to the receive FIFOs of the nodes. 


For scan operations, the node values are combined cumulatively — that is, the 
result for each node is the combination of the values transmitted by all nodes 
having lower (or higher) relative addresses. Forward scans combine values in 
order of ascending node addresses. Backward scans combine values in order of 
descending node addresses. 

Reduction is a special case of scanning. When a reduction message is sent, the 
values from all participating nodes are combined into a single value, and then 
this single result is sent to all the nodes. 


The legal combiner and pattern values for scans and reductions can be specified 
as symbolic constants. The combiner argument must be one of the constants 


ADDJ3CAN 
UADDJSCAN 
OR_SCAN 
XOR_SCAN 
MAX SCAN 


Signed addition. 
Unsigned addition. 
Bitwise inclusive OR. 
Bitwise exclusive OR. 
Signed maximum. 


and the pattern argument must be one of the constants 

■ scanjforwakd Values are combined in ascending address order. 

■ s can_backward Values are combined in descending address order. 

■ scan_reduce Reduction operation. 


Important: If you are sending a message that is longer than one word, the order 
in which the words of the message are written depends on the combine operation: 

■ Maximum operations require the most significant word to be written first. 

■ Both types of addition require the least significant word to be written first. 

■ Inclusive and exclusive OR have no word-ordering requirement. 
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Scanning with Segments 

You can use segmented scanning to divide a partition into segments of nodes — 
regions of nodes within which forward and backward sc anning is done indepen¬ 
dently of all other nodes in the partition. The scan values obtained within each 
segment do not affect the values obtained in any other segment. 

Note: Reduction operations do not use segmented sc annin g. Reduction scans 
ignore the current segment settings. 

The following control register is used to read and set the current segmentation: 

n±_scan_start One-bit control register, indicates start of scan segments. 

The one-bit flag in ni_scan_start is used to indicate the starting points of 
segments. Segments begin in each node where ni_scan_s tar t is 1, and extend 
through the nodes in order of node address — upward for forward scans, down¬ 
ward for backward scans. If no ni_scan_start flags are set in a partition, then 
die entire partition is treated as one segment. 

Note: It is an error to change the value of the scan_start flag while the com¬ 
bine said FIFO is not empty. (For example, you can’t toggle the scan_start 
flag in the middle of a series of pipelined combine operations.) 

You can read and modify the value of ni_scan_start by using these macros: 

int value = CMNA_segment_start(); 
CMHA_set_segnient_start (value) 

Important: If you are sending a multiword message, the value of 
ni_scan_start when the first value is written applies to the entire message. 
Altering the flag after the first value is written has no effect on the message. 


Addition Scan Overflow 

Addition scans on large values can cause arithmetic overflow in some nodes. The 
ni_com_scan_over£low flag in the status register indicates whether the 
current scan result has suffered arithmetic overflow. 

ni_com_s t atus Status register. 

ni_com__scan_overf low Hag, set if add scan had overflow. 
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This flag is 1 if the current message in the receive FIFO suffered arithmetic over¬ 
flow; otherwise, it is 0. You can obtain the current value of this flag by using the 
field extraction macro: 

value = COMBINE_OVERFLOW (status) ; 

Note: The com_scan__overf low flag’s value is defined only when the current 
message in the receive FIFO is the result of a scan or reduction operation with 
a combiner of addition or unsigned addition. 

You can also instruct the NI to signal an interrupt for scan overflow. The 
private register contains a flag, ni_com_scan_overf low_ie, that when set 
to 1 causes an a Green interrupt (scan overflow) to be signaled when a scan 
result that overflowed is read from ni com recv. 


4.2.9 Network-Done Messages 

Network-done messages are used to synchronize the processing nodes after a 
Data Network operation. A network-done message is sent by a node when it has 
completed sending its Data Network messages and is waiting for the other nodes 
to finish. (Of course, even after a node has sent a network-done message, it may 
still receive Data Network messages.) 

Important: Although network-done messages are directly related to the opera¬ 
tion of the Data Network, they are a feature of the combine interface of the 
Control Network. All non-abstaining processors must start a network-done mes¬ 
sage before the network-done operation can be completed. 

A network-done message is always of length 1, and the actual word written is 
ignored — all that matters is the sending of the message itself. Network-done 
messages have a unique pair of combiner and pattern values: the combiner field 
for the message must be 5, and the pattern field must be 0. 

There is a unique pair of combiner and pattern constants that are used to signal 
a network-done operation: 

combiner: assertjrouter_done pattern: 3 can_router_done 

Network-done messages are an exception to the usual message-reception inter¬ 
face of the combine interface. The result of a network-done message is not 
delivered as a value in the receive FIFO. 


NI Version 2.2 (CM-5E), June 1994 

Copyright © 1994 Thinking Machines Corporation 


79 



NI Programmer’s Handbook 


Instead, the Data Network flag ni_router_done_comp 1 ete is used to indi¬ 
cate when the network-done message has been sent by all nodes: 

nl_dr_s t a tus Data Network (DR) status register. 

nl_router_done_complete Network-done completion flag. 

When a node sends a network-done message, the ni_router_done_complete 
flag of that node is set to 0. When all non-abstaining nodes have sent a network- 
done message, and when the Data Network has no pending messages for any 
node, the n±_router_done_complete flag is set to 1 for all nodes. 

You can use the following C macro to access this flag: 

DR_ROUTER_DONE ( Status) 

Usage Note: An attempt to send a network-done message with a length other 
than 1, or to send a network-done message while another such message is still 
in progress (that is, while the ni_router_done_coraple te flag is zero) signals 
a Bus Error. 


How Network-Done Works... 

Network-done messages continually use the combine interface hardware until 
they are completed, so any combine operations started after a network-done 
won’t complete until after the network-done message is completed. 

The network-done operation makes use of the ni_dr_message_count register 
of the Data Network to determine when the Data Network is clear. As described 
in Section 3.5.4, each node increments this register when it sends a message, and 
decrements the register when it receives a message. (Not counting, of course, 
messages for which counting is disabled by a 0 flag in ni_count_mask.) 

When the n±_dr_message_count register is zero for all non-abst aining nodes, 
there should be no messages in transit through the Data Network. (Again, this 
may not be the case if there are messages for which message-counting is dis¬ 
abled, but this does not prevent the use of the network-done operation.) 

A network-done message basically does a repeated addition scan on the values 
of the ni_dr_message_count register for all non-abstaining nodes. When the 
global result of this scan is zero, then the NI assumes that the Data Network is 
clear, and sets the ni_router_done_complete flag to 1. 
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...And Why You Should Care 

Since network-done operations involve a combine interface scan of the value of 
a Data Network register, you should be careful about setting and changing the 
abstain flags of the combine interface when you intend to send a network-done 
message. (See Section 4.2.10 for a discussion of the combine interface’s abstain 
flags.) 

For example, if you change the combine abstain flags of one or more nodes while 
a Data Network operation is in progress, you may inadvertently exclude one or 
more nodes that have non-zero message_count registers. If you then start a 
network-done operation, these registers are ignored by the implied addition scan. 
In most cases, this prevents the result of the scan from ever becoming zero, and 
thus prevents the network-done message from completing. 

To send a network-done message safely, make sure that the combine abstain flags 
of all nodes that might send or receive a message via the Data Network are 
cleared before starting the Data Network operation, and make sure those abstain 
flags remain cleared until after the network-done message has been completed. 


4.2.10 Abstaining from the Combine interface 

The combine interface has two abstain flags that you can use to cause the NI to 
abstain from combine interface transactions. 

n±_com_control Status register, contains combine abstain flags. 

ni_rec_abstain Flag, combine interface abstain flag. 

ni_reduce_rec_abstain Flag, special reduction abstain flag. 

Setting the ni_rec_ai>sta±n flag to 1 causes the NI to discard any arriving 
combine interface messages, and allows any messages sent by other nodes to 
complete without the participation of the abstaining node. 

In the case of combine operations that expect a value from each node, abstaining 
nodes effectively supply an appropriate identity value for the operation. 
However, no result value is written to an abstaining node’s receive queue (with 
the exception of reduction operations — see below). 
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You can use the abstain flag macros described in Section 2.6.3 to read and write 
the abstain flag, using the register address constant com_control_reg: 

value = CMNA_read_abstain_flag(com_control_reg); 
CMNA_write_abstain_flag(com_control_reg, value) ; 

Important: As with all abstain flags, the ni_rec_abstain flag and the 
ni_reduce_rec_abstain flag should be changed only when there are no 
messages pending in the combine interface. If a message is currently being writ¬ 
ten to the send FIFO when either abstain flag is changed, a Yellow interrupt (com 
abstain changed) is signaled. 

Implementation Note: Because of way the broadcast and combine interfaces 
interact, a node that is abstaining from a combine operation should not execute 
a broadcast operation until the combine operation is completed. (For more in¬ 
formation, see Section 8.2.6.) 


The Reduction Receive Abstain Flag 

For scan operations, no result value is written to an abstaining node’s receive 
FIFO. For reduction operations, however, there is an additional abstain flag, 
ni_reduce_rec_abstain, that controls whether or not the abstaining node 
receives the result. 

Setting this flag to 1 causes a node to ignore the results of reduction operations. 
If n±_rec_abstain is 1 and ni_reduce_rec_absta±n is 0, the node 
receives the results of reduction operations without having to supply a value for 
them. (For more detail, see the section on reduction operations below.) 

You can use the following macros to read and write the receive abstain flag: 

value = CMNA_read_rec_abstain_f lag (com_con.trol_reg) ; 
CMNA_wr ite_xec_abstain_f lag (com_con.tr ol_r eg, value) ; 

For the Curious: The reason for this distinction is that there are important cases 
where it is necessary for a node to receive the result of a reduction without having 
to participate in it. For example, when you want to transfer a value from the 
nodes of a partition to the partition manager, you can set the combine abstain 
flags so that the nodes transmit a reduction message and only the PM receives it. 
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4.2.11 The Combine Private Register 

The combine interface’s private register contains the following subfields: 


n±_com_pr I vat e 
ni_rec_ok_ie 
ni_lock 
ni_rec_stop 
ni_rec_full 
ni_com_s can_over £ low_ 
ni_com_rec_empty_ie 
ni_c om_a end_leng th 
ni__coai_send_combiner 
ni_com_aend_pattarn 
ni com send start 


/ate register. 

Hag, “Receive OK” interrupt enable. 
Interface lock flag. 

Interface stop flag. 

Flag, indicates receive FIFO is full. 
Flag, scan overflow interrupt enable. 
Flag, empty rec. FIFO inter, enable. 
Field, send-message length. 

Field, send message combine value. 
Field, send message pattern value. 
Flag, scan segmentation flag. 


The rec_ok_ie, lock, rec_stop, and rec_£ull subfields are as described 
in Chapter 2. The ni_com_scan_over £ 1 ow_ie flag is described in Section 
4.2.8. The remaining fields are described in the sections below. 


Empty Receive FIFO Interrupt 

When the ni_com_rec_empty_ie flag is set to 1, the NI si gnals a Green inter¬ 
rupt (com rec empty) if the receive FIFO ever becomes empty (that is, when 
the rec_ok flag becomes 0). This allows the supervisor to insert one or more 
messages into the empty receive FIFO, so that from a user program’s point of 
view, the FIFO is never empty. (This is used by the OS in context switching.) 


Clearing the Combine Send FIFO 

The pipelining feature of the combine interface means that when the supervisor 
needs to swap a process out, there may be several complete messages pending 
in the combine send FIFO, each of which has its own auxiliary information (each 
message may have different combine and pattern values, for instance). 

The supervisor extracts messages from the send FIFO by reading them, one at a 
time, from the ni_com_send register. Reading a value from this register extracts 
the word (or doubleword) that was most recently pushed into the FIFO. 

Important: Once the supervisor begins reading words from the send FIFO, the 
FIFO must be emptied before a new message can be written to it. (This avoids 
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the potential for accidentally pushing a new message on top of a half-extracted 
old message.) The effect of violating this restriction is undefined. 


Usage Note: It is only legal to read a value from the ni_com_send register 
when the combine interface is not being used (that is, when the receive FIFO is 
empty and no node in the partition is or will be in the process of writing a com¬ 
bine message while the contents of the send FIFO are being read out. 

The four private register fields send_length, send_comb±ner, 
send_pattern, and send_start contain the auxiliary data and segmentation 
information for the most recent message in the send FIFO (that is, the message 
that includes the next word that the supervisor can read from the send FIFO). 


Specifically: 

ni_com_send_length 
ni_com_send_comb±ner 
ni_com_send_pattern 
ni com send start 


Field, send message length. 

Field, send message combine value. 
Field, send message pattern value. 
Flag, scan segmentation flag. 


■ send_length is the number of words in the entire message. 


■ send_combiner is the combine value for the message. 


■ send_j?attern is the pattern value. 

■ send_start is the ni_scan_start register value for the message. 


The supervisor can use these fields like the corresponding status register fields 
to obtain the auxiliary data for messages extracted from the send FIFO. The 
send_length field is undefined for a network-done message. (The message is 
always one word in length.) The value of scan_start is undefined for reduc¬ 
tion and network-done messages, which ignore the segmentation fla g. 
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4.2.12 Combine Interface Examples 

The examples shown here are fragments of code that are intended to be run on 
the processing nodes. See Chapter 7 for a discussion of large-scale program 
structure. 


Sending and Receiving a Combine Message 

This function sends a message via the combine interface. The message is as¬ 
sumed to be composed of length words of data starting at the location specified 
by message, and is sent with the given combiner and pattern. 

int COM_send(combiner, pattern, message, length) 
int ‘message, combiner, pattern, length; 

{ int i, start, step; 

/* For max scans, send high-order word(s) first */ 
if (combiner==MAX_SCAN) {start=length-l; step=-l;} 
else { start=0; step=l; } 

CMNA_com_send_first(combiner, pattern, 

length, message[start]); 
for (i=l; i<length; i++) 

CMNA_com_send_word(message[(start+=step)]); 
return(SEND_OK(CMNA_com_status())); } 

This function receives a message, stores it in memory beginning at the location 
specified by message, and returns the length of the message received. (Note that 
a combiner must also be specified, so that maximum scans are retrieved in the 
right order.) 

int COM_receive(combiner, message) 
int ‘message; 

{ int i, length, start, step; 

while(!RECEIVE_OK (CMNA_com_status())) {} 

length=RECEIVE_LENGTH(CMNA_com_status()); 

/‘For max scans,receive high-order word(s) first*/ 
if (combiner==MAX_SCAN) {start=length-l; step=-l;} 
else { start=0; step=l; } 
for(i=0; iclength; i++) { 

message [start] = CMNA_com_receive_word(); 
start+=step; } 
return(length); 

} 
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Executing Scans and Reduction Scans 

This function sends and receives a scan using the given message of length words, 
with the specified combiner and pattern, storing the result in memory starting at 
result. 

int COM_scan(combiner, pattern, message, 
length, result) 

int *message, *result, combiner, pattern, length; 

{ 

int status=0, rec_length; 
while (!status) 

status=COM_send(combiner,pattern,message,length); 
rec_length = COM_receive(combiner,result); 
return(rec_length); 

} 

Here’s an example of a simple scan using integer values: 

int send[MAX_COMBINE_MSG_WORDS], 

receive[MAX_COMBINE_MSG_WORDS]; 

for (i=l; i<MAX_COMBINE_MSG_WORDS; i++) 
send[i]=i; 

COM_s c an(ADD_SCAN, SCAN_FORWARD, send, 

MAX_COMBINE_MSG_WORDS, receive); 

As a practical example, you can use a reduction scan on integer values to get the 
number of non-abstaining processors in the current partition: 

int send = 1, receive = 0; 

COM_scan (ADD_SCAN, SCAN_REDUCE, ScSend, 1, ^receive) ; 

printf("Actual number of processors: %d\n", 
CMNA_partition_size ) ; 

printf("Scanned number of processors: %d\n", 
receive ); 
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Executing a Network-Done Operation 

Here’s a simple network-done synchronizing function: 

void network_done_synch() 

{ 

CMNA_com_s end_fir s t(ASSERT_ROUTER_DONE, 

SCAN_ROUTER_DONE,1,0); 

while (!DR_ROUTER_DONE(CMNA_dr_s tatus())) {}; 

For example: 

int message - 1; 

int network_done_msg = 0; 

int next_processor = (CMNA_self_address+l) 

% CMNA_partition_size; 

/* Send a message */ 

LDR_send (next_processor, &message, 1, 0); 

/* Synchronize the nodes */ 
network_done_synch() 

/* Retrieve the message */ 

LDR_receive (&message, 1); 
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4.3 The Global Interface 

The global interface provides a generic synchronization mechanism for the 
CM-5’s processing nodes. It is much like the network-done feature of the com¬ 
bine interface, but without the additional condition that the Data Network must 
be clear before the operation can complete. 

The global interface combines a single bit from every participating node in a 
logical OR operation, and then returns the result to each node. The actual values 
sent by the nodes, however, can be completely arbitrary. The sending of the mes¬ 
sage itself is sufficient to provide synchronization of the nodes. 

A global interface message can be sent by one of three subinterfaces: 

* the synchronous global interface, which requires that all nodes send a 
message before any receive the result 

■ the asynchronous global interface, which permits nodes to send a message 
and read the result at any time, with the network continually monitoring 
the state of all participating nodes 

■ the supervisor asynchronous global interface, which is identical to the 
asynchronous global interface save that its registers are accessible only 
from the supervisor area 

There is a separate register set for each of these three methods. Each of these 
interfaces is described in more detail in the sections below. 


The Global Interface Registers at a Glance: 


hex offset 





ni sync crlobal send 

OxOOCO 


ni_hodgepodge 

0x00B8 


ni_aBync_sup_global 

OxOOBO 


ni_async_global 

0X00A8 


SiW-SSSSic' irgmUiX*. ySS 


ni_sync_global__abstaln 

0x0098 


ni_sync_global 

0x0090 



Figure 16. NI registers associated with the global interface. 
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4.3.1 The Three Global Register Interfaces 

Unlike the broadcast and combine interfaces, the global interface does not use 
the generic interface model presented in Chapter 2. The following registers are 
used for the three interfaces: 

Synchronous global interface: 
n±__sync_global_send 
ni_sync_global_abstain 
ni_sync_global 
n±_hodgepodge 

Aynchronous global interface: 

ni_async_global Asynchronous send and receive flags. 

n±_hodgepodge Contains interrupt enable flag. 

Supervisor aynchronous global interface: 

ni_async_sup_global Supervisor asynch. send and receive fla gs 

ni_hodgepodge Contains interrupt enable flag. 

The purpose and use of these registers is described in the sections below, and 
Figure 16 contains a memory map showing their relative locations in NI 
memory. 


4.3.2 The Synchronous Global Interface 

The synchronous global interface takes the global OR of a flag set by <»a ch node. 
Each non-abstaining node must set its synchronous global flag (and thereby sp.nH 
a synchronous global message) before the result of the operation is reported to 
any node. 

The following registers and flags form the synchronous global interface: 

n±_sync_global_send Used to send the first value of a message. 
ni_sync_global_abstain Used to abstain from synch, global msgs. 

ni_sync_global Used to receive a message. 

ni_sync_global_rec Synchronous global receive flag. 

ni_sync_global_complete Synchronous global completion fla g 

n±_hodgepodge Contains interrupt enable flag. 

ni_sync_global_rec_ie Receive interrupt enable flag. 


Used to send the first value of a message. 
Used to abstain from synch global msgs. 
Used to receive a message. 

Contains interrupt enable flag. 
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Sending and Receiving Messages 

To start a synchronous global interface message, write a value (either 0 or 1) to 
the the ni_sync_global_send register. To do this, use the macro 

CMNA_or_global_sync_bit {value) 

When you write a value to the global_send register, the ni_sync_glob- 
al_complete flag is set to 0, indicating that a message is in progress. (Note: 
It is an error to write to the ni_sync_global_send register when the 
ni_sync_global_complete flag is 0.) 

When all participating nodes have sent a message, the global interface takes the 
logical OR of the ni_sync_gl obal_s end flag in each node, and then sets the 
ni_sync_global_rec flag of every participating node to the result. At the 
same time, the ni_Bync_global complete flag is set back to 1 to indicate 
completion of the message. To detect when the message has completed and to 
retrieve the resulting global value, use the macros 

value = CMNA_global_sync_coinplete () ; 
value * CMNA_global_sync_rec(); 


Abstaining from Synchronous Global Messages 

The synchronous global interface includes an abstain flag that can be used to 
exclude a node from the interface’s operations: 

nl_sync_global_abstain Status register, contains global abstain flag. 

When the ni_sync_global_abstain flag is set to 1, synchronous global mes¬ 
sages complete without the node’s participation (as if the node has sent the 
message with its ni_sync_global_send flag set to 0). You can use the abstain 
flag operations described in Chapter 2 to read and write the value of the 
ni_sync_global_abstain register. (The address constant for this register is 
sync_global_absta±n_reg.) For example: 

value=CMNA__read_abstain_f lag (sync_global_abstain_reg) ; 
CMNA_write_abstain_f lag (sync__global_abstain_reg, value) ; 

Note: As with all abstain flags, ni_sync_global_abs tain should be changed 
only when there is no global message pending. A Bus Error is signaled if the 
abstain flag is modified when the ni_sync_global_complete flag is 0. Also, 
a Bus Error is signaled if the ni_sync_global_send register is written while 
the abstain flag is 1. 
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Synchronous Global Receive Interrupt 

If the ni_sync_global_rec_ie flag in the hodgepodge register is set to 1, 
then a Green interrupt (sync global rec) is signaled whenever the 
ni_sync_global_rec flag changes from 0 to 1. 


Supervisor Operations for the Synchronous Global Interface 

The supervisor can write a new value into the ni_sync_global_r ec flag when 
the flag ni_sync_global_complete is set to 1. If ni_sync_global_rec is 
written when nl_sync_global_complete is 0, a Bus Error 
(bad memory access) is signaled. 

Implementation Note: Even when ni_sync_global_rec_ie is 1, the super¬ 
visor’s writing a 1 to ni_sync_global_rec does not signal the corresponding 
Green interrupt (sync global rec). 

The supervisor can take control of the synchronous global interface (for exam¬ 
ple, during a context-switch) as follows. Each node in the partition to be 
context-switched should save the values of the sync_global_complete, 
sync_global_rec and sync_global_send flags. All nodes for which 
ni_sync_global__complete is 1 should write a 1 to the 
ni__sync_global_send flag, thus completing any pending operation. 

To restore the state of the synchronous global interface, all nodes restore the 
value of the sync_global_send flag by writing the saved value back into it. 
When the resulting global operation completes (ni_sync_gl obal_compl ete 
becomes 1), all nodes restore the saved value of the ni_sync_global_rec 
flag. All nodes with a saved value of 0 for n±_sync_global_complete write 
the ni_sync_global_send flag again to restart the interrupted global opera¬ 
tion. Control can then be handed back to user code. 


4.3.3 The Asynchronous Global Interface 

The asynchronous global interface is not so much a node synchronization tool as 
a means for determining whether all the nodes are still operating properly, or 
whether some global action needs to be taken. As with the synchronous interface, 
the asynchronous interface takes the global OR of a flag set by each node. How¬ 
ever, this global OR is performed repeatedly, so that a change of a flag by any 
node is reported almost immediately to the other nodes. 
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For example, each node can set its flag to 1 before performing an operation, and 
set the flag to 0 when the operation is completed. The global interface returns a 
1 value until all nodes have set their flags to 0, guaranteeing that all nodes have 
completed the operation. 

The following registers and flags form the asynchronous global interface: 

ni_async_global Control register, contains the following flags: 
ni_global_send Flag, used to “send” asynchronous messages. 
ni_global_rec Flag, always set to logical OR of send flags. 


ni_hodgepodge 

ni_global_rec_ie 


Control register, includes the following flag: 
Flag, global receive interrupt enable. 


Sending and Receiving Messages 

Because the asynchronous global interface operates continuously, there really is 
no such thing as “sending” or “receiving” a message via this interface. 

The ni_global_rec flag in each node is continually updated to reflect the 
“current” logical OR of the ni_global_send flag in all nodes. When any node 
writes a new value into its ni_global_send flag, the change is propagated to 
the ni_global_rec flag of all other nodes after a brief interval. 

Important: Because this is an asynchronous mechanism, the ni_global__rec 
flag may not always reflect the present state of the ni_global_send flags in 
all the nodes. There is always a delay between the instant any node changes its 
n±_global_send flag and the instant that all nodes receive the result of the 
change. You should not write code that depends on this delay having any exact 
length, but you can assume that the delay is no longer than the time taken to 
transmit a synchronous message. 

To set the value of the nl_global_send flag, use the macro 
CMNA_or_global_async_b 11 {value) ; 
and to retrieve the value of the n±_global_rec flag, use the macro 
value = CMNAqlobalasvncread(); 
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Asynchronous Global Receive Interrupt 

If the n±_global_rec_ie flag in the hodgepodge register is set to 1, then a 
Green interrupt (global rec) is signaled whenever the ni_global_rec flag 
changes from 0 to 1. 


4.3.4 The Supervisor Asynchronous Global Interface 

The supervisor asynchronous global interface is identical to the asynchronous 
interface described above, except that its registers are accessible only from the 
supervisor area. This interface is typically used by the operating system to syn¬ 
chronize the nodes during OS operations such as context switching. 

For example, if each node sets its flag to 0, then the global interface returns a 0 
value until one of the nodes signals a 1 instead. If any node reaches a point in 
its operations where OS intervention is required, the node can set its flag to 1, 
signaling a 1 value to all the other nodes, and also indicating to the OS that some 
global action must be taken. 

The following register and flags form the supervisor asynchronous interface: 

ni_async_sup_global Control register, contains these flags: 

ni_supervisor_global__send Flag, used to “send” messages. 
ni_superv±sor_global_rec Flag, logical OR of send flags. 

ni_hodgepodge Control register, includes the flag: 

nl_supe rvl a o r_gl obal_r e c_ie Supervisor receive interrupt enable. 


Sending and Receiving Messages 

The ni_supervisor_global_send and ni_supervisor_global_rec 
flags are used to send and receive messages the same way that the asynchronous 
interface does (described above). 


Supervisor Asynchronous Global Receive Interrupt 

If the n±_supervisor_global_rec_ie flag in the hodgepodge register is 
set to 1, then a Green interrupt (supervisor global rec) is signaled when¬ 
ever the ni_supervisor_global_rec flag changes from 0 to 1. 
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4.3.5 Global Interface Examples 

The examples shown here are fragments of code intended to be run on the pro¬ 
cessing nodes. See Chapter 7 for a discussion of large-scale program structure. 


Using the Synchronous Global Interface 

Here’s a function that executes a simple barrier synchronization using the global 
interface. 

int global_sync_value(value) 
unsigned int value; 

{ 

CMNA_or_global_sync_bit (value) ; 

while (!CMNA_global_sync_complete()) {}; 

return(CMNA_global_sync_xead()); 

} 

All non-abstaining nodes must execute this function for the global message to be 
completed. If you don’t need to send or receive a value, you can rewrite this as 

int global_sync() 

{ 

CMNA_or_global_sync_bit (1) ; 

while (!CMNA_global_sync_complete()) {}; 

(void) CMNA_global_sync_read(); 


Using the Asynchronous Global Interface 

The following function sends a value using the asynchronous global interface, 
and then immediately reads and returns the current value from the receive regis¬ 
ter: 


int CMNA_global_async(value) 
unsigned int value; 

{ 

CMNA_or_global_async_bit(value); 
return (CMNA_global_async_read()); 

} 
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NI Interrupts 


The NI chip is, in many ways, the “interrupt gateway” of the CM-5. Most node 
hardware and software exceptions, whether or not they originate in the NI chip, 
are signaled to the node microprocessor via NI interrupts. 

The NI is capable of signaling an interrupt in any of five classes and at any of 
a number of levels of severity. Interrupts can be signaled by events beyond the 
programmers’s control (such as hardware failures), or by fatal errors in the way 
a program uses the NI, or deliberately, under program control. 

Interrupts are signaled by one of two different methods: 

■ as a local interrupt to the NTs associated microprocessor 

■ as a broadcast interrupt to the other NIs in the partition 

This chapter describes the kinds of interrupts available on the NI, their causes, 
the registers used to determine their type and severity when they are signaled, 
and the mechanism used to signal a broadcast interrupt. 


5.1 Interrupt Classes 

The NI can signal five different classes of interrupt: Red, Orange, Yellow, Green, 
and Bus Errors. Red interrupts tend to be the most severe and Green interrupts 
the least severe. The five types are distinguished as follows: 

■ Red interrupts indicate a failure of the hardware, such as checksum vio¬ 
lations and message format errors. 
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They occur at unpredictable times relative to the instruction stream and are 
usually irrecoverable. Determining the precise cause of a Red interrupt 
may require the use of the Diagnostic Network. 


The possible Red interrupts are: 

internal fault 
dr checksum error 
cn checksum error 
cn hard error 
me error 
emu error 
be interrupt red 


Failure detected in NI chip itself. 
Data Network checksum failure. 
Control Network checksum failure. 
Control Network hardware failure. 
Error detected in memory subsystem. 
Cache/MMU error. 

Red broadcast interrupt. 


Orange interrupts indicate that the attention of the operating system is 
required, as in timer interrupts and broadcast interrupt messages. 

They occur at unpredictable times relative to the instruction stream and do 
not destroy any information that might be needed to determine the cause 
of the interrupt. 

The possible Orange interrupts are: 

timer interrupt NI timer reached interrupt_now. 

rdone complete Router done complete interrupt, 

be interrupt orange Orange broadcast interrupt. 


Yellow interrupts indicate that the software has made an error. They are 
usually irrecoverable, as they indicate that your program is doing some¬ 
thing illegal and must be rewritten. Sufficient information is retained in 
the NI to permit isolation of the cause of the interrupt, but it is not always 
possible to recover all the information relating to the cause of the interrupt. 

Yellow interrupts are associated with particular instructions, but usually 
are not signaled at the exact point of the offending instruction, because of 
the loose coupling between the NI and the microprocessor. 

The possible Yellow interrupts are: 


dr count negative 
be or com collision 
com abstain changed 
bad relative address 
bad memory access 
message too long 
be interrupt yellow 


Negative DR message count. 
Conflict in broadcast/combine ops. 
Rag changed while interface in use. 
Address outside partition, etc. 

Bus Error signaled as interrupt. 

Data Network message too long. 
Yellow broadcast interrupt. 
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. ■ Green interrupts indicate the occurrence of common events for which 
the software has requested notification, such as the arrival of messages, 
the si gnalin g of broadcast interrupts, arithmetic overflow in a scan, etc. 
There is one interrupt for each event, and each event’s interrupt can be 
enabled and disabled independently under the control of the supervisor. 

Depending on the type of event, the interrupt may or may not occur syn¬ 
chronously with a particular instruction. No information is lost by a Green 
interrupt. 


The possible Green interrupts are: 

scan overflow 
dr/ldz/rdr rec ok 
bc/sbc rec ok 
sbc rec ok 
com rec ok 
com rec empty 
dr/ldr/rdr rec tag 
ldr/rdr user rec tag 
dr rec all fall down 
sync global rec 
global rec 

supervisor global rec 
dperr 

sfifo empty 
be Interrupt green 


Overflow in combine interface scan. 
DR/LDR/RDR message received. 
Broadcast received. 

Supervisor broadcast received. 
Combine message received. 

Empty combine receive FIFO. 
Message with interrupt tag received. 
LDR/RDR interrupt tag received. 
All Fall Down message received. 
Synchronous global msg received. 
Asynchronous global msg received. 
Supervisor asynch. msg received. 
Vector unit error. 

Data Network send FIFO empty. 
Green broadcast interrupt. 


■ Bus Errors indicate that a bus transaction cannot be completed, as in an 
attempt to read an address that does not correspond to a register, or to write 
a message that does not conform to sending protocol (send_f lrst, then 
send). Bus Errors are signaled asynchronously, and are irrecoverable. 

There is basically one flavor of Bus Error: 

bad memory access Meaningless or illegal reference. 

Bus Errors are treated differently from the four colored interrupts. Bus 
Errors are always handled as traps, primarily because they occur only on 
read operations, and do not involve the NI chip. 

Note: Bus Errors are distinct from segmentation violation errors. Seg¬ 
mentation errors result from attempting to read an unmapped virtual 
address, and are signaled synchronously with the offending instruction. 
Bus Errors result from errors with physical addresses, once the address has 
been transmitted to the Mbus itself. 
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.1 Disabling Bus Errors 

Some Mbus devices do not respond well to the NI signaling a bus error. In order 
to allow the NI to be used in systems that include such devices, the NI can 
optionally “signal” a bus error as a Yellow interrupt (bad_memory_access). 
This feature is controlled by a flag in the n±_hodgepodge register, 
ni_disable_bus_error. This flag is turned off by default, and by an NI reset, 
to provide backward compatibility. 


.2 Interrupt Pathways 

The four colored interrupts (Red, Orange, Yellow, and Green) result from a num¬ 
ber of different causes. Figure 17 shows the pathways followed by the various 
types of interrupts on their way to the microprocessor. These pathways are 
described in detail in the sections below. 
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Figure 17. The possible pathways for colored interrupts. 
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5.2.1 Red Interrupts 

The Red interrupts are of two varieties: 

■ On-chip faults — Har dware errors detected by the NI itself. 

■ Off-chip faults — Problems on other devices that are signaled via the NI. 

On-chip faults are universally fatal — that is, they always cause the OS to halt 
(usually forcefully). It is then necessary to use diagnostic measures to determine 
the cause of the problem. 

Off-chip faults are caused by problems on other components, and it is necessary 
for the OS to poll those devices to find out what happened. 

Of the red interrupts, the following are off-chip faults: 

me error — Error in MC (memory controller). 

emu error — Error in CMU (cache and memory unit). 

The cause of these faults can only be determined by examining the state of the 
appropriate hardware: 

■ MC errors are caused by either a fault in the MC itself (usually fatal), or 
(if the CM-5 has the vector unit option installed) by an error signaled from 
one or more of the vector units. In either case, it is necessary to examine 
the state of the appropriate hardware to determine the actual cause of the 
interrupt. 

■ CMU errors are only caused by bad memory writes (typically memory 
writes to illegal addresses) and are always fatal. CMU errors are asynchro¬ 
nous, so that the error is not signaled until some time after the offending 
write instruction. 

All the remaining Red interrupts are on-chip faults. Three are caused by network 
problems: 

dr checksum error — Data Network fault. 

ca checksum error — Control Network fault. 

cn hard error — Control Network hardware fault. 

One is caused by NI chip problems: 

internal fault — NI chip fault. 
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And one can be signaled by software: 

be interrupt red — Red broadcast interrupt. 

Warning: A Red broadcast interrupt is functionally equivalent to deliberately 
causing a fatal error, so use it with caution — if you use it at all! 


5.2.2 Orange Interrupts 

There are three Orange interrupts. One is caused by the NI timer: 

timer interrupt — Timer alarm interrupt. 

One is caused by the completion of a Data Network network-done operation: 

rdone complete — Network-done-complete interrupt. 

And the remaining interrupt can be signaled by software: 
be interrupt orange — Orange broadcast interrupt. 


5.2.3 Yellow Interrupts 


The Yellow interrupts are, with one exception (the Yellow broadcast interrupt), 
caused by NI violations produced in user code: 


com abstain changed 
be or com collision 
bad relative address 
dr count negative 
bad memory access 
message too long 


— Illegal abstain flag change. 

— Multiple message collision. 

— Illegal DR destination. 

— Negative DR message count. 

— Bus Error signaled as interrupt. 

— Data Network message too long. 


There is also a Yellow broadcast interrupt that can be signaled by software: 


be interrupt yellow 


— Yellow broadcast interrupt. 
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5.2.4 Green Interrupts 


The Green interrupts are, for the most part, indications of non-error network 
events for which the user may want to assign a specific code handler. 

For example, there are nine Green interrupts, one for each major network inter¬ 
face, that indicate when a message has arrived in the interface’s recv register: 


— BC interface message received. 

— SBC interface message received. 

— COM interface message received. 

— DR interface message received. 

— LDR supervisor message received. 

— RDR supervisor message received. 

— Asynchronous global message received. 

— Synchronous global message received. 


be rec ok 
sbe rec ok 
com rec ok 
dr rec ok 
ldr rec ok 
rdr rec ok 
global rec 
sync global rec 
supervisor global rec — Supervisor asynchronous global message. 


In addition, there is a Green interrupt for an important combine interface event: 


scan overflow — Combine interface add-scan overflow. 


There are a number of interrupts for OS-related events: 


dr rec tag 

ldr rec tag 

rdr rec tag 

ldr user rec tag 

rdr user rec tag 

dr rec all fall down 

com rec empty 

dperr 

sfifo empty 


— DR message arrived with interrupting tag. 

— LDR message arrived with interrupting tag. 

— RDR message arrived with interrupting tag. 

— LDR message with user interrupt tag received. 

— RDR message with user interrupt tag received. 

— DR All Fall Down mode message received. 

— Combine receive FIFO empty. 

— Vector unit error. 

— Data Network send FIFO empty. 


And as usual there is a broadcast interrupt that can be signaled by software: 


be interrupt green — Green broadcast interrupt. 
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5.3 The interrupt Cause and Clear Registers 


There are six 141 registers that you can use to determine which interrupt(s) have 
been signaled, to clear the interrupts once you have finished handling them, and 
to force interrupts to be signaled when necessary: 


ni_interrupt_cause 

ni_interrupt_cause_green 

ni_inte-rrupt__clear 

ni_interrupt_clear_green 

ni_interrupt_set 

ni_inte r rup t_s e t_gr een 


Flags set by non-Green interrupts. 

Flags set by Green interrupts. 

Flags used to clear non-Green interrupts. 
Flags used to clear Green interrupts. 
Flags used to set non-Green interrupts. 
Flags used to set Green interrupts. 


When an event causing an interrupt occurs, a bit in the ni_interrupt_cause 
or ni_interrupt_cause_green register is set. Which bit is set indicates what 
the event was. If more than one interrupt occurs before any are cleared, several 
bits in these registers may be set simultaneously. 


Interrupts can be cleared by writing a value to the ni_interrupt_clear or 
ni_interrupt_clear_green registers. Any value written to these registers 
should contain ones in locations corresponding to the interrupts that are to be 
cleared. It is not possible to read the value of the ni_interrupt_clear or 
nl_interrupt_clear_green registers — use the corresponding cause reg¬ 
ister to determine whether any interrupts have not yet been cleared. 

Note: If a given interrupt has an interrupt enable flag (a flag with a name ending 
in _ie) and the flag is set to 0, then the interrupt is not signaled and the corre¬ 
sponding n±_interrupt_cause or ni_interrupt_cause_green flag is 
not set. 


Interrupts can be triggered artificially by writing to either of the 
ni_interrupt_set or ni_interrupt_set_green registers. The value 
written to the register should contain one bits in locations corresponding to the 
interrupts that are to be signaled. In the case of an interrupt with an enable bit, 
the interrupt can be signaled even if the interrupt is currently disabled. It is not 
possible to read the ni_interrupt__clear, ni_interrupt_clear_green, 
ni_interrupt_set or ni__interrupt_set_green registers. 

The ±nterrupt_cause and interrupt_cause_green registers may also be 
written explicitly (by the supervisor, not user code) to cause interrupts to be sig¬ 
naled without their usual triggering event occurring. 
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5.4 Interrupt Levels 


Each of the four color classes of interrupt includes a “level” or “priority” value 
that can be used to provide the software with information about the relative im¬ 
portance or priority of interrupts of various colors. 


Any interrupt level can be assigned to each color of interrupt. It is, for example, 
permissible to give Green interrupts a level of 15 while Red interrupts have a 
level of 4. However, the relative interrupt levels are intended to indicate priority 
or severity; for example, there are mechanisms for masking all interrupts (of any 
color) below a given level. 

The following register is used to set the priority value for each interrupt color: 


ni_int er rup t_l eve 1 

ni_interru.pt_level_red 
n±_interrupt_level_orange 
ni_interrupt_level_yellow 
ni_interrup t_leve l_green 


Control register, contains these fields: 
Red interrupt priority level. 
Orange interrupt priority level. 
Yellow interrupt priority level. 
Green interrupt priority level. 


The four eight-bit fields, level red through level green, each indicate the 
level at which the corresponding color of interrupt is signaled. For example, if 
the level red field is set to 13, all red interrupts from that point onwards are 
signaled to the microprocessor with a level of 13. 

If more than one color of interrupt is signaled simultaneously, the interrupt level 
signaled to the processor is the inclusive OR of the levels for each interrupt color. 


If any of the inter rupt_leve 1 fields is set to 0, then all interrupts of the corre¬ 
sponding color are suppressed. (When the NI is reset, for example, all four 
interrupt level fields are set to 0.) 


Implementation Note: Currently, only the low-order bit of each interrupt level 
field is used. The other bits are required to be 0. 
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5.5 Broadcast Interrupts 

The broadcast interrupt mechanism allows an interrupt to be signaled from one 
NI to all other NIs in the current partition. Each NI receiving the broadcast im¬ 
mediately signals an interrupt to its associated microprocessor. 

Important: Only one NI in each partition can use the broadcast interrupt facility. 
If two or more NIs try to broadcast simultaneously in the same partition, a Yel¬ 
low interrupt (be or com collision.) is signaled to all nodes in the 
partition, and the broadcast interrupt messages that are received are undefined. 

The broadcast interrupt can be of any color, Red, Orange, Yellow, or Green. A 
unique flag exists in the cause, clear, and set registers for each color of 
broadcast interrupt. Only Bus Errors cannot be broadcast — mainly because it 
is not useful (and doesn’t really make sense) to do so. 

The following register and flags are used to send a broadcast interrupt: 

ni_interrupt_send Register used to send broadcast interrupt. 

ni_Jhodgepodge Control register, includes the flags: 

ni_interrupt_send__ok Flag, set when broadcast is sent. 

ni_interrupt_rec_enable Flag, enables receipt of interrupts. 

To send a broadcast interrupt, write a value to the ni_interrupt_send register 
indicating the color of interrupt to be signaled. The permissible values for each 
color of interrupt are as follows: 


Value 

Interrupt 


Description 

8 

be 

Interrupt 

red 

Red broadcast interrupt. 

4 

be 

interrupt 

orange 

Orange broadcast interrupt. 

2 

be 

interrupt 

yellow 

Yellow broadcast interrupt. 

1 

be 

interrupt 

green 

Green broadcast interrupt. 


Note: More than one color of interrupt can be broadcast at a time (for example, 
by combining the above values with a logical-OR operation). Multi-colored 
broadcast interrupts are signaled by the hardware exactly as if each colored inter¬ 
rupt was signaled separately. The software effects of such multi-colored 
interrupts are determined entirely by the current interrupt handlers on the nodes. 

Writing a value to ni_interrupt_send sets the ni_interrupt_send_ok 
flag to 0 until the interrupt has been successfully broadcast, at which point the 
flag is set back to 1. An attempt to write a value to ni_interrupt_send while 
ni_interrupt_send_ok is 0 signals a Bus Error. 
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Any NI can disable broadcast interrupts by setting its ni_interrupt_rec_en - 
able flag to 0. Doing so causes all broadcast interrupts received by that NI chip 
to be ignored. Setting the flag back to 1 re-enables broadcast interrupts. 

Note: There is a special class of broadcast interrupt, the Reset interrupt, which 
c anno t be disabled. See Section 6.10 for more information about the cause and 
effects of an NI Reset. 


5.6 Recovering from Interrupts 

The methods used to recover from an interrupt depend heavily on the type of 
interrupt itself. Appendix B of this manual provides guidelines describing the 
steps needed to recover from each of the possible interrupts. 
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Other NI Interfaces and Features 


This chapter describes the remaining NI registers and features (those not covered 
in the preceding chapters). Except as noted, all registers and features described 
in this chapter are accessible only to the supervisor. 


6.1 The “Hodgepodge” Register 


The ni_hodgepodge register, as its name suggests, contains a collection of mis¬ 
cellaneous flags that are used by various features of the NI. 


n±_h.odgepodge 

ni_sync_global_rec_ie 
n±_global_rec_ie 
n±_superv±sor_global_rec_ie 
ni_interrupt_send_ok 
ni_interrupt_rec_enable 
ni_flush_complete 
n±_tixner_ie 

n±_configuration_complete 

ni_cn_stop_send 

ni_d±sable_bus_error 

ni_ldr_r ec_tag_ie 

ni_r dr_r e c_t ag_i e 

ni_ldr_user_rec_tag_ie 

ni_rdr_u.ser_rec_tag_±e 

ni_msg_too_long_ie 


Register with “hodgepodge” of flags: 
Sync global receive interrupt enable. 
Asynch global receive intrpt. enable. 
Supervisor asynch. rec. intrpt. enable. 
Broadcast interrupt send ok flag. 
Broadcast interrupt receive enable. 
Combine flush complete flag. 

NI timer interrupt enable flag. 
Configuration complete flag. 

Control Network disable flag. 

Bus Error disable flag. 

LDR supervisor tag interrupt enable. 
RDR supervisor tag interrupt enable. 
LDR user tag interrupt enable. 

RDR user tag interrupt enable. 
Message too long interrupt enable. 
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For more information on the meaning and use of these flags, refer to the sections 
describing the NI features that use them. (Look up the individual flags by name 
in the Index.) 


6.2 Node Address Registers 

There are three NI registers that provide information about the physical address 
of the current node within the CM-5, as well as the size and location of the cur¬ 
rent partition: 

ni_physical_self 20-bit physical address of current node. 

ni_partition_base 20-bit address of first node in partition. 

ni_partition_size Number of nodes in current partition. 

These registers are used by other NI chip features, such as the chunk table 
address translation mechanism described in Section 6.3 below. 


6.3 NI Chunk Table and Address Translation 

The NI chunk table is a small array stored in the NI itself that determines the 
locations of the “chunks” of processing nodes that make up a Data Network 
partition on the CM-5. A chunk is a contiguous sequence of physical addresses 
that correspond to real, working processing nodes. Addresses corresponding to 
broken or missing hardware are isolated by not being included in any chunk. 

Important: The chunk table specifies chunks of node addresses — the chunk 
table has nothing to do with memory allocation on the nodes. 


6.3.1 Node Address Translation 

The chunk table is used to convert from relative node addresses used within a 
partition to the physical addresses required by the Data Network. 

For the Curious: A side effect of the use of the chunk table is that it implicitly 
divides the Data Network up into “partitions” of nodes. That is, there is no hard- 
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ware restriction preventing a Data Network message from traveling between 
partitions; it is the chunk tables that determine whether a relative address is legal 
for a given partition of nodes. 

The mapping from relative to physical addresses is performed in three steps: 

First, the relative address is compared with the ni_partition_size register, 
to determine whether it is legal for the current partition. (If the relative address 
is greater than or equal to n±_partit±on_size, the address is guaranteed not 
to correspond to a node in the current partition, and an error is signaled.) 

Next, the relative address is split into two parts (see Figure 18). 



Figure 18. Translation from relative addresses to physical addresses. 


The two parts of the address are: 

the high-order bits of the address, known as the chunk address 
the low-order bits of the address, known as the select address 
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The chunk address is used as a pointer into the NTs chunk table. The referenced 
chunk table entry, known as the chunk position , is recombined with the select 
address to form an absolute address — essentially an offset from the address of 
the first processor in the current partition. 

Finally, the absolute processor address is added to the value of the register 
ni_partition_base to get the required physical address. 


6.3.2 Chunk Sizes and Address Allocation 

The size of the chunk table is determined by the number of bits in a ch unk ad¬ 
dress (call this a), and the number of bits in a chunk position (call this p). The 
chunk table consists of 2 a entries, each p bits long. The values of a and p are 
currently fixed by hardware at a = 6 and p = 8. Thus, the ch unk table contains 
64 entries, each 8 bits long. 

However, while the size of the chunk table is fixed, the size of the chunks it refer¬ 
ences (that is, the number of physical addresses per chunk) is under supervisor 
control. The following register is used to set the chunk size: 

ni_chunk_s±ze Size of chunks referenced by the chunk table. 

The ni_chunk_s±ze register contains a three-bit value that determines the 
number of bits in the select address part of a relative address, and thus sets the 
number of addresses per chunk. 

The number of bits in a select address is 2 ni _ chunk _ size . As a result, the number 
of physical addresses in a chunk is 4 ni - chunk _ Bize , and this means that the num¬ 
ber of possible relative addresses (in other words, the number of accessible 
nodes) is 2 a * 4 nl _ chun k_ siz ®. This also means that the total physical address 
space accessible through the chunk table is 2P * 4 ni_chunk_size. Thus, the acces¬ 
sible physical address space is always 2P~ a times the size of the relative address 
space. This extra “unused” space between chunks is used to isolate regions of 
broken or missing hardware. (See Figure 19.) 
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Figure 19. The chunk table is used to map contiguous relative addresses 
onto discontiguous physical addresses. 


In the simplest case, the chunk table is set up to map all relative addresses to a 
contiguous region of 2 a * 4»i_ clIuak _ aiste physical addresses. In this case, chunk 
table entry n simply has the value n. 

The table below lists the permissible values for the ni_chunk_s ize register, 
along with the corresponding number of relative addresses (nodes) per chunk, 
and the maximum size of the physical address space in nodes and addresses. 


ni_chunk_size 

Addresses/chunk 

Nodes 

Phys. address space 

1 

4 

256 

IK 

2 

16 

IK 

4K 

3 

64 

4K 

16K 

4 

256 

16K 

64K 

5 

IK 

64K 

256K 

6 

4K 

256K 

1M 


Note: The effects of writing ni_chunk_size with a value not listed in this table 
are undefined, but almost certainly disastrous. 
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6.3.3 Modifying the Chunk Table 

The following registers are used to read and write chunk table entries: 

ni_chunk_table_data Location used to read/write table entries. 
ni_chunk_table_address Chunk table location that is read/written. 

Note: The chunk table is set up by the OS when the nodes are grouped into parti¬ 
tions, and from then on the chunk table is normally not modified. Accordingly, 
the registers listed above are accessible only from the supervisor area. 

When the ni_chunk_t ab 1 e_da t a register is written, the value written is 
stored in the chunk table entry indicated by ni_chunk_table_address. "When 
the table_data register is read, the value read is the current contents of that 
chunk table entry. 

The ni_chunk_table_address register determines the entry of the chunk 
table that is affected by reading or writing the ni_chunk_tai>le_data regis¬ 
ter. The size of the values that are read from and written to this register depends 
on the size of chunk addresses (see the discussion in Section 6.3.2). 

Important: In order for the Control Network to operate correctly, the entries of 
the chunk table must be in ascending order. In other words, each chunk table 
entry must contain a larger address than the entry that precedes it. 

Note: The effects of reading or writing the table_data register while the Data 
Network is in use are undefined, and best avoided. 


6.4 Combine Interface Flush 

The combine interface flush operation is used to reset the hardware of the com¬ 
bine interface, canceling any uncompleted combine operations. As with all other 
Control Network operations, a combine flush must be started in unison by all of 
the nodes in a partition — nodes cannot “abstain” from a flush. Also, flushes 
only affect the single partition in which they are started; they don’t cross partition 
boundaries. 

Important: The broadcast and global interfaces are not affected by flushing, and 
must be cleared separately. 
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The combine flush interface consists of the following registers and flags: 

ni_com_f lush_send Single-flag register used to start a flush. 

ni_hodgepodge Control register, includes the flag: 

ni_f lush_complete Flag, set when flush is completed. 

To start a flush operation, write a value (either 0 or 1, the actual value is unimpor¬ 
tant) to the ni_com_£lush_send register. This sets ni_f lush_complete to 
0, and then starts the interface flush. When the flush is completed, the 
f lush_comp 1 ete flag is set back to 1. Attempting to write the 
ni_com_£lush_send register while nl_flush_complete is 0 or 
ni_com_aJbsta±n is 1 signals a Bus Error. 

Important: A flush operation should be executed only when there are no mes¬ 
sages in transit through the combine interface, that is, when 
n±_com_send_empty is 1, and n±_com_rec_ok is 0. 

Usage Note: The combine flush operation is useful only when the send and 
receive FIFOs of the combine interface are empty. The combine flush operation 
does not clear out the FIFOs — it merely resets the communications hardware of 
the interface itself. The flush operation is only intended to be used in context 
switches, after the FIFOs have been cleared and saved. 


6.5 The NI Timer 


The NI contains a simple timing mechanism that can be used to measure the time 
between two events and to interrupt the microprocessor after a specific interval. 

The following registers and flags form the timer interface: 


n±_t±me 

ni_interrupt_now 
ni_hodgepodge 
nl timer ie 


Timer register, regularly incremented. 
Register, timer value that triggers interrupt. 
Control register, includes the flag: 

Timer interrupt enable flag. 


The 32-bit register ni_time contains an unsigned value that is incremented at 
every microprocessor clock cycle. When the timer value exceeds the register’s 
capacity, it wraps around to 0. 


The value of the ni_time register can be read at any time, and can be written 
by the supervisor to set the NT’s timer to a chosen value. 
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The NI timer can signal an interrupt at a specific timer value. When the value of 
n±_time equals the value stored in the ni_±nterrupt_now register, an Orange 
interrupt (timer interrupt) is signaled. 

This interrupt can be enabled and disabled by setting the ni_t±mer_ie flag in 
the hodgepodge register. When this flag is 1, timer interrupts are enabled. When 
this flag is 0, timer interrupts are disabled. 


6.6 The Bad Address Register 

When a Bus Error is signaled as the result of an illegal memory reference, the 
ni_bad_address register contains the illegal address, the data size, and the 
type (read or write) of the transaction. The data returned by a read from an illegal 
memory address is undefined. Data written to an illegal memory address is lost. 

ni_bad_address Bad address register, contains the fields: 

ni_bad_addxess_low Low 20 bits of illegal address. 

ni_bad_addzess_type Size and type of transaction. 

Usage Note: The ni_bad_addxess register is updated every time a memory 
transaction is made, not just when an error occurs. Thus, its value is valid only 
when a Bus Error (ni bad memory access) has actually been signaled. If 
more than one illegal access is performed before the first one is handled, the val¬ 
ue of the ni_bad_address register is the most recent bad memory address. 

Currently, the format of the ni_bad_address_type field is 


31 29 28 27 26 24 23 20 

rrrvn -■- 1 - 1 - 1 - 1 -■- 1 -■- 1 -'- 



pins 

lok 

csh 

size 

type 


where 

type indicates the transaction type (0 = write, 1 = read) 

size gives the data size (2 - word, 3 = doubleword) 

csh, lok are the MBUS cacheable and lock bits 

pins is the setting of the NTs two physical base address pins 

Values for the type and size fields other than those shown above are reserved. The 
csh, lok, and pins fields are hardware-related and not useful to NI programmers. 
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6.7 NI Partition Configuration 

The NI has a register that can be used to change the partitioning of the CM-5. 
The following register and flag are used to control the partitioning feature: 

ni_conf igruration Partition configuration control register. 

ni_hodgepodge Control register, includes the flag: 

ni_configuration_complete Flag, set when partitioning is done. 

The ni_con£iguration is a five-bit register that controls the configuration, or 
set of processor partitions, that is in use. The value in this register is actually the 
“height” (number of layers) of the Control Network partition to which the node 
belongs. Control Network operations use this value to determine the maximum 
height of the network to which a message needs to be sent. 

By writing a value to the configuration register, you can temporarily change 
the size of the current partition. (Since the actual size of the partition is currently 
determined by the state of the Control Network itself, you can only reduce the 
size of the partition.) 

Note: Only one NI per partition needs to write a value to the configuration 
register — the configuration operation includes all nodes in the same partition. 

The actual value written to the ni_conf iguration register is an encoded ver¬ 
sion of the new partition size: 

configuration = log2( partition_size ) + 2 

Extra for Experts: By writing a 0 to the configuration register, you can tem¬ 
porarily isolate each node in the partition in its own “mini-partition,” so that 
network operations performed by each node apply only to that node. Obviously, 
you should restore the original value of the configuration register when you 
are finished using this “mini-partition” effect 

The flag ni_conf iguration_complete is set to 0 while the repartitioning is 
in progress, and then set back to 1 to indicate its completion. At the same time, 
the ni_conf iguration register of the NI that sent the message is updated to 
the new partitioning value. The configuration registers and flags of the other NIs 
are not affected. An attempt to write a value to the ni_conf iguration register 
while ni_con£iguration_complete is 0 signals a Bus Error. 

Important: A partition change should not be done when the Control Network 
is in use — the effect of doing so is undefined, but certainly disastrous. 
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Disabling the Control Network 

There is one last flag in the hodgepodge register that has not yet been described: 

ni_hodgepodge Control register, includes the flag: 

ni_cn_s top_send Flag, disables Control Network sending. 

This flag is used to completely disable the Control Network, preventing any mes¬ 
sages from being sent into it — including the periodic “idle” packets that are sent 
when the network is not otherwise being used. 

The stop_send flag is generally used only during an NI Reset (see Section 
6.10) when it is necessary to totally disable the Control Network. When the 
stop_send flag is 1, the Control Network is disabled. When the stop_send 
flag is set to 0, normal network operations resume. 

For the Curious: The Control Network is designed in such a way that packets 
are periodically sent into it even when the network is not in use. When no mes¬ 
sage is being sent by the user or by the OS, these “idle” packets simply contain 
no data, and have no effect on the nodes. However, idle packets can affect the 
state of the Control Network itself in unwelcome ways, especially during a Reset 
operation, when it is important for the state of the network to remain unchanged. 

For the Even More Curious: Because the Data Network operates in an essen¬ 
tially asynchronous manner, with messages being sent from the nodes “on 
demand,” the Data Network does not transmit idle packets, and thus has nothin g 
analogous to the Control Network’s stop_send flag. 


NI Serial Number 

Finally, one NI register contains the hardware serial number of the NI chip: 

ni_serial_number Version serial number of NI chip. 

This serial number identifies the version of NI chip that is install ed. 

Usage Note: Most revisions of the NI chip do not have usefully distinguishable 
serial numbers, so this register is not particularly valuable. 
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6.10 NI Reset 

Under the following conditions, the NI chip is completely reset: 

* The system administrator requests a repartitioning of the CM-5. 

■ The system administrator uses the diagnostic hardware of the CM-5 to 
reset the processing nodes and networks. 

When the NI is reset, a number of its register fields and flags are set to known 
states. The following events occur on an NI Reset: 

■ ni_disableJbus_error is negated. 

■ ni_longest_dr_message is set to a value of 5. 

■ All abstain and lock flags are set to 1, thus isolating the NI from all net¬ 
works. These flags are: 

n±_dr_lock ni_ldr_lock ni__rdr_lock 

ni_bc_lock ni_sbc_lock ni_com_lock 

ni_reduce_rec_abstain ni_com_abstain 
ni_bc__r ec_abs tain n±_sbc_r ec_abs tain 

ni_sync_global_abstain 

■ ni_interrupt_level is set to 0. This disables all colored interrupts. 

■ All sending and receiving FIFOs are cleared. 

■ ni_f lush_complate and ni_sync_global_complete are set to 1. 

The values of all other NI registers are undefined, and must be set by software. 

NI Reset is triggered by a special broadcast interrupt, the Reset interrupt, that can 
be sent from another NI or from the partition manager. This interrupt is always 
effective and cannot be disabled. 
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Writing NI Programs 


In this chapter we’ll start applying some of the tools presented in the preceding 
chapters. First, we’ll cover important small-scale issues, such as exchanging data 
between the nodes in a partition and the partition manager. Next, we’ll look at 
a short program that makes use of every network interface of the NI. 


7.1 Transferring Data between Nodes and the PM 

As described in Section 3.3, each node in a partition has a unique address based 
on its location in the partition. However, the PM is not part of this addressing 
scheme. The PM is always located outside of the address space of the partition 
that it manages: 



Figure 20. The partition manager stands apart from the partition it manages. 


This means that sending messages to and from the partition manager involves 
some careful coordination between the PM and the nodes. 
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7.1.1 Sending Messages from the PM to Nodes 

To send a message from the PM to a node, the PM does two broadcast operations: 
one to send the address of the node that should “receive” the message, and one 
to transmit the message itself. 

For example: 

void PM_send_to_NODE(node_address, value) 
int node_address, value; 

{ 

CMNA__bc_s end_f i r s t (1, node_addr ess) ; 
CMNA_bc_send_fixst (1, value); 

} 

Each of the nodes should perform two broadcast reads, one to determine whether 
the address of the message matches the node’s own address, and one to either 
receive and store the message or to ignore it, based on the supplied node address: 

int NODE_ge t_fr om_PM(de s t) 
int *dest; 

{ 

int address, value; 

while (!RECEIVE_OK(CMNA_bc_status())) {}; 

address=CMNA_bc_receive_word(); 
while (!RECEIVE_OK(CMNA_bc_status())) { } ; 

value=CMNA_bc_receive_word(); 

if (address == CMNA_self_address) *dest=value; 

} 

Notice that the node waits until the rec__ok flag is set each time it tries to receive 
a value from the broadcast interface. This is important — while these routines 
are written so that the PM’s two broadcast values should arrive in the node’s re¬ 
ceive queue nearly simultaneously, it’s still necessary to check the rec_ok flag 
before each broadcast read, because the two values are still separate messages. 

Also, notice that in this example only one node “accepts” the value sent from the 
PM, but there’s no reason why you can’t have more than one node “accept” the 
value — you can use any test you like to decide whether the nodes keep or dis¬ 
card the values they receive. 
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7.1.2 Sending Messages from Nodes to the PM 

Sending a message from a node to the PM is almost as straightforward, but in¬ 
volves two interfaces this time: broadcast and combine. 

First, the PM sets its ni_com_abstain flag to 1 and its ni_reduce_rec_ab- 
stain flag to 0, so that it can receive a combine message without having to send 
a value. (Note: We’ll handle this step separately in Section 7.2, below.) 

Next, the PM broadcasts a message containing the address of a processing node, 
as in the PM_send_to_NODE example above. The nodes respond by si gnalin g 
a combine message (a uadd_scan reduction), in which only the node with the 
address specified by the PM transmits a value. (The other nodes supply an identi¬ 
ty value of 0 for the reduction.) The PM then receives this message to get the 
requested value. 

Here’s the function that handles the PM side of this transaction: 

int PM_get_from_NODE(node_address) 
int node_address; 

{ 

CMNA_bc_send_first(1, node_address); 
while (!RECEIVE_OK(CMNA_com_s tatus())) {} ; 
return(CMNA_com_receive_word() ); 

I 

And here’s the corresponding node function: 

void NODE_send_to_PM (value) 
int value; 

{ 

int address; 

while (!RECEIVE_OK(CMNA_bc_status())) {}; 

address = CMNA_bc_receive_word(); 
if (address != CMNA_self_address) value = 0; 

CMNA_c om_s end_first(UADD_SCAN,SCAN_REDUCE,1,value); 
while (!RECEIVE_OK(CMNA_com_status())) {}; 

(void) CMNA_com_receive_word(); 

} 

Notice that immediately after the nodes send a combine message, they perform 
a combine read to discard the resulting value. You might think it would be a good 
idea to temporarily toggle the combine abstain flags for the nodes, so that they 
will simply ignore the result. However, this is not such a good strategy. (Why 
not? See Section 7.2.) 
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7.1.3 Signaling the PM 

Because the above PM/node communication functions use both die broadcast and 
combine interfaces, we’ll want a function that forces the PM to wait until the 
nodes have finished their computations before the PM broadcasts a request for 
the results. A single function will suffice for both the PM and the nodes: 

void PM_NODE_synch() 

{ 

CMNA_or_global_sync_bit (1) ; 

while (!CMNA_global_sync_complete()) {}; 

(void) CMNA_global_sync_read {) ; 

} 

This function uses the global interface to create a simple barrier synchronization. 


7.1.4 For the Curious: Using the Data Network 

You can also use the Data Network to send messages between the partition man¬ 
ager and the nodes. However, owing to the distinction between addressing on the 
nodes and on the partition manager, it’s not as clear-cut an operation as using the 
broadcast and combine methods described above. 

To send a message from the partition manager to a specific node via the Data 
Network, you can use the methods presented in Chapter 3, using the node’s ad¬ 
dress as the destination for the message. 

To send a message from a node to the partition manager, however, you must 
make a system function call: 

int *source, length, tag 

CMNA interface send packet to scalar ( source, length, tag) 

where the interface abbreviation is dr, Idr, or rdr, depending on the network 
interface you wish to use, and the other arguments are as noted in Chapter 3. The 
partition manager can then receive this message as usual. There is a catch, how¬ 
ever — this system call is currently implemented as a trap instruction, which in 
practical terms means it is much less efficient than the combine network method 
shown in Section 7.1.2. 

Sending messages to and from the PM via the Data Network is primarily useful 
when you want to send a message to a specific node without requiring all the 
other nodes to stop and do a network operation at the same time. 
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7.2 Setting the Abstain Flags 

Both the PM and the nodes will need to modify their abstain flags in order to use 
the above functions. Since they will also need to restore the previous values of 
these flags afterwards, it makes sense to use a single pair of functions to handle 
saving and restoring the flags, rather than individually toggling flags within a 
program. 

Also, while changing abstain flags in the middle of a program does work, it’s 
error-prone because it requires that you ensure die corresponding network(s) are 
empty before changing the abstain flag settings. It’s much more straightforward 
to simply set the abstain flags appropriately at the beginning of your program, 
and then leave them alone as much as possible. 

With these factors in mind, here is a pair of functions that handle saving and 
restoring the abstain flags, giving them whatever intermediate settings you se¬ 
lect. 

First, a routine that saves the current values of the abstain flags and then sets 
them to new values: 


int bc_abstain_flag, 
com_abstain_flag, 
com_r ec_abs tain_flag, 
sync_global_abstain_f lag ; 

void save_and_set_abstain_flags 

(new_bc, new_com, new_com__r ec, new_sync_global) 
int new_bc, new_com, new_com_rec, new_sync_global; 

{ 

bc_abstain_flag = 

CMNA_read_abstain_flag(bc_control_reg); 
com_abstain_flag = 

CMNA_read_abstain_flag(com_control_reg); 
com_rec_abstain_flag = 

CMNA_read_rec_abstain_flag(com_contxol_reg); 
sync_global_abstain_flag = 

CMNA_read_abstain_f lag (sync_global_abstain_reg) ; 
CMNA_write_abstain_flag(bc_control_reg, new_bc); 
CMNA_write_abstain_flag (com_control_reg, new_com) ; 
CMNA_write_r ec_abstain_flag(com_contr ol_r eg, 

new_com_rec) ; 

CMNA_wr i t e_abs t ain_f lag (sync_global_abs tain__r eg, 
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Next, a function that restores the old values: 


void restore_abstain_flags() 
{ 


} 


CMNA_write_abstain_flag(bc_control_reg, 

bc_abstain_flag); 

CMNA_wr i te_abs tain_f lag {coin_contr ol_r eg, 

com_abstain_flag); 

CMNA_wr i te_r ec_abstain_f lag (com_contr ol_r eg, 

com_rec_abstairx_f lag) ; 
CMNA_write_abstain_flag(sync_global_abstain_reg, 

sync_global_abstain_flag) ; 


One caveat about these functions: they assume that none of the Control interfaces 
are in use when you call them. This should be the case if you call them at the 
beginning and end of your program, as they are intended to be used. If you need 
to use functions like these within the body of a program, you should precede and 
follow them with code (function calls, etc.) that synchronizes the nodes, thus en¬ 
suring that none of the affected interfaces are in use. 

For example, you can use the global interface to synchronize the nodes while you 
change the abstain flags for the other intrafaces, and then use the network-done 
operation of the combine interface to synchronize while you change the abstain 
flags for the global interface. (You can probably now see why it’s easier just to 
set these flags once and then ignore them!) 


7.3 Broadcast Enabling 

Along with setting the abstain flags, there’s one other important operation that 
needs to be included in any NI program. As noted in Section 4.1.8, you need to 
call the macro 

CMNA_par ticipate__in (NI_BC_SEND_ENABLE) ; 

to enable broadcast sending — even if you clear the broadcast abstain flag. The 
best point in your program to do this is the same place you set the abstain flags. 
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7.4 NI Program Structure 

Now, with these tools we can turn to the task of designing an NI program. 

An NI program consists of three files: 

■ code to be run on the partition manager 

■ code to be run on the nodes (one program executed by all nodes) 

■ an interface file de finin g the node routines that are callable from the PM 

The sections below describe each of these parts in detail, and show you how to 
bring them together into a working program. 


7.4.1 The cmna.h Header File 

Important: Both the partition manager code file and the node code file must 
#include the header file cmna. h, as follows: 

#include <cm/cmna.h> 

This header file contains #include directives that load the other files needed to 
define the NI programming tools described in this manual. Note: If you plan to 
call CMOS_signal () (see Section 3.S.4), you must also #include the header 
file <cmsys/cai_signal .h>.) 


7.4.2 Partition Manager Code 

Code that runs on the PM may contain anything ordinarily included in a program 
running on a Sun computer. This includes prlntf calls, system calls, I/O calls, 
and calls to other specialized libraries. The simplest PM program might look 
something like this: 

#include <cm/cmna.h> 
void mainO { 

/* start node program running */ 
node_program(); } 

This program does nothing more than call the corresponding node program de¬ 
fined below. Typically, however, the PM code will include operations that send 
data to the nodes and retrieve the results of the node computations. 
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7.4.3 Node Code 

Code written for execution on the nodes consists of one or more subroutines that 
perform local computations and make NI calls to send messages through the net¬ 
works. Node programs can also include simple I/O calls to display intermediate 
results. 

In particular, the output of printf calls from all nodes is collected and saved 
in a file (typically named “CMTSD_pr intf . pn .pid") that you can examine dur¬ 
ing and/or after execution of your program. However, the handling of printf 
calls from the nodes slows down program execution considerably, so this method 
of output is best used only for debugging your program. 

Note: As of this release, many UNIX system calls are not supported on the 
nodes. If node programs invoke these unsupported calls, segmentation violations 
may be signaled. You should use node subroutines primarily for computations 
and NI operations, and use the PM code for system calls and external I/O. 


The Node’s “Main” Routine 

The First subroutine in the node file must be the one initially called by the PM. 
This routine serves much the same function as the “main” routine in standard C 
programming — it is the trigger that starts everything else running. 

While you can give a node subroutine any name that you wish, if it is to be called 
from the PM, then you must add the prefix cmpe_ to the subroutine name when 
defining it and when calling it from another node subroutine. This prefix is used 
by the compiler to determine which subroutines will be called from the PM. You 
do not have to use the cmpe_ prefix anywhere outside of the node subroutine file. 

The simplest node program, corresponding to the PM program given above, is 

#include <cm/cmna.h> 
void CMPE_node_program() { 

/* Node program, does nothing, just an entry point 
*/ 

} 

As you can see, this is less than the bare bones of a subroutine — it does nothing 
at all. We’ll see an example of a complete node program below. 
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7.4.4 Interface Code 

The “interface code” file is nothing more than a file of function prototypes, as 
might appear in a header file. It is used in the compilation process to produce 
special declaration code that allows the nodes to respond correctly to subroutine 
calls from the PM. 

The interface code file for the skeletal program given above has just one line: 

void node_program(); 

Important: Before you compile it, the interface code file must be preprocessed 
by the utility program sp-pe-stubs. This utility program translates your inter¬ 
face prototypes into complete subroutine calls that can be compiled with the PM 
and node code files to produce an executable NI program. 

This is the reason that node functions callable from the PM require the cmpe_ 
prefix — the sp-pe-stubs utility adds this prefix to the name of each host- 
callable function, so that there’s no possibility of collision with names of node 
functions that you have not defined as host entry-points. 


7.5 A Sample Program 

As an example, here’s a simple NI program that uses each of the CM-5 network 
interfaces. First, the partition manager source file: 

Filename: Ni_test. c 

/* Sample NI program - PM program */ 

#include <cm/cmna.h> 

#include "utils.h" 

void main () { 

int input, result, high_node; 

printf("\nSimple NI test program, by W.R.Swanson,\n"); 
printf("Thinking Machines Corporation—1/31/92.\n\n"); 

/* Enable broadcast sending */ 

CMNA_participate_in (NI_BC_SEND_ENABLE) ; 
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/*Abstain from broadcast reception, combine sending */ 
save_and_set_abstain_flags(1,1,0,0); 

/* Start node programs running */ 
node_main () ; 

/* Get value from the user, send it to the nodes. */ 
printfC'This CM-5 partition has %d nodes.\n", 
CMNA_partition_size); 

printf ("Please type an integer to send: ")’; 
scanf("%d", &input); 

PM_send_to_NODE(0, input); 

printfC'Sent value %d to node 0 ... \n" , input) ; 

/* Wait for the nodes to finish juggling numbers */ 
PM_NODE_synch(); 

/* Get value from high-address node */ 

/* (size - 2, because scan result starts with 0) */ 
high_node = CMNA_par tition_siz e-2; 

result * PM_get_from_NODE(high__node); 
printfC'Got value %d (should be %d) from node %d.\n", 
result, input, high_node); 
result = PM_ge t_f r om_N0DE(0); 

printfC'Got value %d (should be %d) from node 0.\n", 
result, (input*(high_node+l)}); 

restore_abstain_flags() ; 

} 

Next, the corresponding code for the processing nodes: 

Filename: Nl_test. node. c 

/* Sample NI program - node program */ 

#include <cm/cmna.h> 

#include "utils.h" 

void CMPE_node_main 0 { 

int value=0, scan_value, flipped_value; 
int mirror__node_addr ; 

CMNA_participate_in (NI_BC_SEND_ENABLE) ; 
save and_set_abstain_flags(0,0,0,0); 


128 


NI Version 2.2 (CM-SE), June 1994 
Copyright © 1994 Thinking Machines Corporation 



Chapter 7. Writing NI Programs 


/* Node 0 gets the value sent by the PM... */ 
NODE_get_f rom_PM (&value) ; 

/* and broadcasts it to all nodes */ 

if (CMNA_self_address==0) CMNA_bc_send_first(1,value); 
while (!RECEIVE_OK(CMNA_bc_status())) {}; 

value = CMNA_bc_receive_word(); 

/* Do an addition scan to put a different value 
in each node */ 

CMNA_c om_s end_first(UADD_SCAN,SCAN_FORWARD,1,value); 
while {!RECEIVE_OK(CMNA_com_status())) {}; 

scan_value = CMNA_com_receive_word(); 

/* Use LDR to "flip" order of values in nodes */ 
mirror_node_addr = 

(CMNA_partition_size-l) - CMNA_self_address; 
CMNA_1dr_send_first(0, 1, mirror_node_addr); 
CMNA_ldr_send_word(scan_value); 
whi le (! RECEIVE_OK (CMNA__ldr_S tatus () ) ) {} ; 

flipped_value = CMNA_ldr_receive_word{); 

/* Signal to PM that answer is ready */ 

PM_NODE_synch(); 

/* Send value from high-order node back to PM */ 
NODE_send_to_PM {f 1 ipped_value) ; 

/* Send value from node 0 back to PM */ 
NODE_send_to_PM (f lipped__value) ; 

restore_abstain_flags(); 

1 


And the interface code file: 

Filename: Ni_test .proto 

/* Sample NI program - interface code */ 
node main(); 
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Finally, both the PM and node programs include a utilities file, which includes 
such tools as the abstain-flag functions and the PM/node communications 
functions: 

Filename: utils,h 

/* Utility code */ 

int bc_abstain_flag, com_abstain_flag, 

com_rec abstain_flag, sync global abstain flag; 


void save_and_set_abstain_flags(new_bc, new_com, 

new_com_r ec, 

new_sync_global) 

int new_bc, new_com, new_com_rec, new_sync_global; 

{ 

bc_abstain_£ lag = 

CMNA_read_abstain_flag(bc_control_reg); 
com_abstain_flag = 

CMNA_read_abstain_flag(com_control_reg); 
com_rec_abstain_flag = 

CMNA_xead_xec_abstain_flag(com_contiol_reg); 
sync_global_abstain_flag = 

CMNA_read_abstain_flag(sync_global_abstain_reg); 


} 


CMNA_write_abstain_flag(bc_control_reg, new_bc); 
CMNA_write_abstain_flag(com_control_reg, new_com); 
CMNA_write_r ec_abs tain_flag(com_contr ol_reg, 

new_com_rec); 

CMNA_write_abs tain_flag(sync_global_abstain_reg, 

new_sync_global); 


void restore_abstain_flags() 

{ 


} 


CMNA_write_abstain_flag(bc_control_reg, 

bc_abstain_flag); 

CMNA_writ e_ab s t a i n_f 1 ag (c om_c ontxol_reg, 

com_abstain_flag); 

CMNA_write_r ec_abs tain_f1ag(c om_c ontrol_reg, 

com_rec_abstain_flag); 
CMNA_wxit e_ab s t ain_f1ag(sync_g1obal_abs tain_r eg, 

sync_global_abstain_f lag) ; 
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void PM_send_to_NODE(node_address, value) 
int node_address, value; 

{ 

CMNA_bc_s end_fir s t(1, node_addr ess); 

CMNA_bc_send_fir s t(1, value) ; 

j 

int NODE_get_from_PM(dest) 
int *dest; 

{ 

int address, value; 

while (!RECEIVE_OK(CMNA_bc_s tatus())) {}; 

address=CMNA_bc_receive_word(); 

while (!RECEIVE_OK(CMNA_bc_status())) {}; 

value=CMNA_bc_receive_word(); 

if (address == CMNA_self_address) *dest=value 

} 

int PM_get_from_NODE(node_address) 
int node_address; 

{ 

CMNA_bc_send_first(1, node_address); 
while (!RECEIVE_OK(CMNA_com_status())) {}; 

return(CMNA_com_receive_word()); } 

void NODE_send_to_PM(value) 
int value; 

{ 

int address; 

while (!RECEIVE_OK(CMNA_bc_status())) {}; 

address = CMNA_bc_receive_word(); 
if (address != CMNA_self_address) value = 0; 
CMNA_com_send_first(UADD_SCAN,SCAN_REDUCE, 

1,value); 

while (!RECEIVE_OK(CMNA_com_status())) {}; 

(void) CMNA_com_receive_word(); 

I 

void PM_NODE_synch() 

{ 

CMNA_or_global_sync_bit (1) ; 

while(!CMNA_global_sync_complete()) {}; 

(void) CMNA_global_sync_read(); 

} 
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7.6 Compiling and Executing an Ni Program 

Note: This section presents a brief overview of the process of compiling and 
executing an NI program. It’s very much like the procedure used in compiling 
and executing a CMMD program — so much so that you should also read the 
CMMD User’s Guide for more information. (In particular, the CMMD User’s 
Guide includes examples of using a generic makefile to compile your code. This 
may be more appropriate to your needs and inclinations than the script example 
shown below.) 

To compile an NI program you must: 

■ preprocess the interface file by calling sp-pe-stubs 

■ compile the resulting file, as well as the PM and node routine files 

* link the three object files together with the CM linking program cmld 

To illustrate this, here are the steps you would take in compiling the sample pro¬ 
gram shown above: 

First, preprocess the interface code file: 

/usr/bin/sp-pe-stubs < NI_test.proto > NI_test.intf.c 

Next, compile the three code files: 

cc NI_test.c -c -g -DCM5 -DMAIN=main 
-1/usr/include 

cc NI_test.node.c -c -g -DCM5 -dalign -Dpe_obj 
-I/usr/include 

cc NI_test.intf.c -c -g -DCM5 -DMAIN=main 
-I/usr/include 

Finally, link everything together. For this purpose, you must use the CM-specific 
linking program cmld: 

/usr/bin/cmld -o NI_test 

NI_test.o NI_test.intf.o 

-L/usr/lib -lcmna_sp -lcmrts —lm 
-pe NI_test.node.o 

-L/usr/lib -icmna_pe -lcmrts_pe -lm 

The result is a single executable file, Nl_test, which you can run by logging 
on to one of the partition managers of a CM-5 and executing the file. 
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7.6.1 A Simple Compiling Script 

Here’s a short UNIX script that automates this process. It takes as its single argu¬ 
ment the name of an NI program, constructs the names of the three component 
files from the program name, compiles the files, and links them together as 
shown above. 


Note: This script assumes that the program files are all present in the current 
directory. 

#! /usr/bin/csh -e -f 

# nicc2 — Compiles an NI program 

echo "Script: $0, Compiling $1 for the NI..." 


set PMFILE = " 

set PMOFILE = " 
set NODEFILE = " 
set NODEOFILE = " 
set INTFFILE = " 
set INTFCFILE = " 
set INTFOFILE = " 
set OUTFILE = " 
set NODEOUTFILE = 
set EXECUTABLE 
set NODEEXECUTABLE 


$l.c" 

$l.o" 

$1.node.c" 

$1.node.o" 

$1.proto" 

$1.intf.c" 

$1.intf.o" 
$ 1 " 

"$l.pn" 

= "a.out" 

= "a.out.pn 


It 


echo 'Preprocessing interface code file: ' $INTFFILE 
/usr/bin/sp-pe-stubs < $INTFFILE > $INTFCFILE 

echo 'Compiling PM code file: ' $PMFILE 

cc -c -g -DCM5 -DMAIN=main -I/usr/include $PMFILE -o 

$PMOFILE 

echo 'Compiling node code file: ' $N0DEFILE 
cc -c -g -Dpe_obj -DPE_CODE -I/usr/include $N0DEFILE 
-O $N0DE0FILE 

echo 'Compiling interface code file: ' $INTFCFILE 
cc -c -g -DCM5 -DMAIN=main -I/usr/include $INTFCFILE 
-o $INTFOFILE 


echo 'Linking it all together...' 

/usr/bin/cmld -lg $PM0FILE $INTFOFILE -o $OUTFILE \ 
-L/usr/lib -lcmna_sp -lm \ 

-pe -lg $NODEOFILE -L/usr/lib -lcmna_pe -lm 
echo 'Done. Executable written to: ' $OUTFILE 
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7.6.2 Compiling and Running the Program 

Note: The following examples assume that you are currently logged in to one of 
the partition managers of a CM-5. 

The output of the compiling script for the NI_test program looks like this: 

% nicc2 NI_test 

Script: nicc2, Compiling NI_test for the NI... 
Preprocessing interface code file: NI_test.proto 
Compiling PM code file: NI_test.c 
Compiling node code file: NI_test.node.c 
Compiling interface code file: Nl_test.intf.c 
Linking it all together... 

Done. Executable written to: NI_test 

The script produces a single executable file Ni_test, which can be executed as 
follows: 

50: NI_test 

Simple NI test program, by W.R.Swanson, 

Thinking Machines Corporation — 1/31/92. 

This CM-5 partition has 32 nodes. 

Please type an integer to send to the nodes: 42 
Sent value 42 to node 0... 

Received value 42 (should be 42) from node 30. 

Received value 1302 (should be 1302) from node 0. 


7.6.3 On-Line Code Examples 

As of Version 7.1.3 of the CM system software, there are on-line copies of the 
sample program and script in this chapter, along with copies of the progr amming 
examples in Appendix C. 

Depending on where your system administrator has chosen to store the CM soft¬ 
ware, these files may be located under the pathname 

/usr/examples/ni-examples 

or they may also be located somewhere else entirely. Check with your system 
administrator for help in locating these files. 
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This chapter presents a number of NI programming issues that you should keep 
in mind, as well as important performance and programming hints and warnings. 

Note: Some of the notes and warnings below are included in earlier chapters. 
They are repeated here so that you can find them quickly. 


8.1 Performance Hints 

8.1.1 NI Register Operation Times 

Here are some rough estimates of the time taken by a number of basic operations: 


register access (register variable): 1 cycle 

cache memory (previously accessed variable): 2-3 cycles 

NI register read (n±_interface_ata.tuB, etc.): 7-8 cycles 

NI register write (nl_inteTface__ status, etc.): 3-4 cycles 

memory access (newly accessed variable): -25 cycles 


The time taken to perform an NI register read or write operation is longer than 
the time taken for cached memory accesses, but much shorter than the time for 
full memory accesses. (NI register writes are faster than reads because an NI read 
operation requires that the node microprocessor wait for the read operation to 
move through the Mbus buffer before a value is actually read and returned.) 

For the Curious: This is why the NI status register tools are designed so that you 
can read an NI status register once and then extract fields from the retrieved val¬ 
ue. Once you have retrieved the value of the NI register and stored it in cached 
memory, the access time for extracting multiple fields decreases substantially. 
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8.1.2 Reading and Writing Registers with Doubieword Values 

While this document focuses for the most part on reading and writing network 
messages in terms of single (32-bit) words, you can also use doubleword (64-bit) 
operations in reading and writing network registers. 

Writing a doubleword to a register has the same effect as writing two singleword 
values, but involves only one register operation. Likewise, reading a doubleword 
from a register is the same as reading two singlewords. 

The combine interface is an exception to this rule, because of its pip elining fea¬ 
ture. You can’t use doubleword writes when you are pipelining combine 
operations. However, you can use doubleword reads with pipelined operations, 
and doubleword writes are permitted for non-pipelined combine operations. 

In addition, attempting a doubleword read or write for a message that consists of 
only one word (as is the case for network-done tests) signals an error. 

For C Programmers: To use doubleword read and write operations, the values 
you send must be doubleword-aligned in memory. To ensure that this is the case, 
use the compiler switch -dalign when compiling any file that includes double- 
word function calls or variable definitions. For example, 

cc -c -g -DCM5 -dalign -I/usr/include ni_code.c 


Example: LOR Send/Receive 

Here’s the LDR_send_rece±ve_msg function from the Data Network chapter, 
rewritten to use doubleword writes: 

int tag_limit = 3; 

LDR_send_receive_msg(dest_address,message,length,tag,dest) 
unsigned dest_address, tag; 
int *message, *dest, length; 

{ 

int send_size, send_size2, receive_size,receive_size2; 
int offset, source_offset=0, dest_o£fset; 
int words_to_send=length, words__received=0 ; 
int packet_size, count, rec_tag, status; 
double *dbl; 

if (((int)message & 3) || ((int)dest & 3)) 

CMPN_jpanic ("Message or dest not doubleword aligned"); 
packet_size = (MAX_ROUTER_MSG_WORDS-1) & -1; 
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while ((words_received < length) || (words_to_send)) { 

/* First try to receive a packet */ 
status=CMNA_ldr_status(); 

if (words_received<length && RECEIVE_OK(status) && 
RECEIVE_TAG(status)<=tag_limit) { 
dest_offset = CMNA_ldr_receive_word() ; 
receive_size = 

RECEIVE_LENGTH_LEFT(CMNA_ldr_status ()); 
for (count=0; count< (receive_size>>l) ; count+ + ) { 

dbl = (double *)(&dest[dest_offset++]); 
dest_offset++; 

*dbl = CMNA_ldr_receive_double () ; 
dbl++; } 

if (receive_size & 1) /* If word left over */ 

dest[dest_offset++] = CMNA_ldr_receive_word(); 
words_received += receive_size; 

} /* if */ 

/* Now try sending a packet */ 
if (words_to_send) { 

send_size = ((words_to_send < packet_size) ? 

words_to_send : packet_size); 
send_size2 = send_size >> 1; 
do { 

CMNA_ldr_send_first(tag,send_size+l, 

dest_address); 

CMNA_ldr_send_word(source_offset); 
offset=source_offset; 

/* Send as many doubles as possible */ 
for (count=0; count<send_size2; count++){ 
dbl = (double *)(&message[offset++]); 
offset++; 

CMNA_ldr_send_dotible (*dbl++) ; } 
if (send_size & 1) /* If a word is left over */ 
CMNA_ldr_send_word(message[offset++]); 

} while {!SEND_OK(CMNA_ldr_status())); 
source_offset=offset ; 
words_to_send -= send_size; 

} /* if */ 

} /* while */ 
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8.1.3 Use Message Discarding for Efficiency 

When a message you are writing to a network send FIFO is discarded, it is com¬ 
pletely discarded — effectively, it is as if you never began writing the message. 

Many NI programmers take advantage of this property by writing a complete 
message to a network FIFO, and only then checking to see whether it was dis¬ 
carded (and if so, writing it again). This might seem a sloppy practice, but it is 
actually a safe and efficient strategy. 

Because messages are typically only a few words long, and because the NI com¬ 
pletely ignores a discarded message, it’s perfectly reasonable to check the 
send_ok flag just once, after you’ve written the entire message. Also, if your 
code is properly written it should be rare for a message to be discarded, and thus 
unlikely that checking the send_ok flag after writing each value of the message 
provides any benefit. In fact, checking the send_ok flag after you write each 
value of a message can slow your code down considerably. 


8.1.4 Set the Abstain Flags Once and Forget Them 

In most cases, abstain flags of a network interface can be changed only when the 
network is not in use — that is, when there are no messages pending in either the 
send or receive FIFOs, and no messages in transit in the network. While this cer¬ 
tainly does not prevent you from toggling the state of the abstain flags within 
your code, it does make this kind of flag-toggling more prone to progr amming 
errors. 

A more straightforward strategy to use is to set the values of the abstain flags 
once, at the beginning of your program, leave them alone while the program 
runs, and then restore their original values before your program exits. 

Note: This last point is important. As noted in Section 2.6.5, some programming 
systems (such as CMMD) use the abstain flags for their own purposes. These sys¬ 
tems are written with the assumption that the abstain flags won’t change 
unexpectedly, so if the flags do change these systems may not operate correctly. 

When you alter the values of the abstain flags, you must take care to save the 
original settings of these flags and to restore them before your code exits. Failing 
to do so can cause your code to signal obscure errors that are hard to trace. 
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8.2 Potential Programming Traps and Snares 

Here are some potential sources of serious errors that you should keep in mind. 

Note: Some of the notes and warnings below are included in earlier chapters. 
They are repeated here so that you can find them quickly. 


8.2.1 Address Calculation on the Partition Manager 

On any of the processing nodes, the ni_base address (the base address of the 
NI register region) is constant, and furthermore is set to a value that is zero in 
all the low-order bits used for the send-first auxiliary data fields. Thus, it is pos¬ 
sible on the nodes to logically IOR the auxiliary data with the base address to get 
the final result. 

On the partition manager, however, the base address of the NI register region can 
be any arbitrary address assigned by the operating system. Hence, on the parti¬ 
tion manager you must use arithmetic addition when combining auxiliary data 
with the base address of a send-first register. 

As a point of style, it is simplest to just use addition in all cases, as this will work 
on either the PM or the nodes. (This is the usage shown in this document.) 


8.2.2 Pay Attention to Data Network Addresses 

When sending a Data Network message with a relative address, the address must 
be valid within the current partition. If an address higher than CMNAjparti- 
tion_size is supplied, the NI signals an error. 

Also, there is currently a 20-bit limit on the length of a Data Network address, 
and the remaining high-order bits in a 32-bit address value must be 0. If any of 
these high-order bits are nonzero, the NI signals a serious error, and in some 
cases the entire partition of nodes may crash. You should either write your code 
so that the high-order bits of a network address can never be other than zero, or 
failing that mask out the top 12 bits of an address before using it. 

Implementation Note: Currently, there is an additional restriction on the most 
significant (19th) bit of the address — it too must be 0, or an error ■will result. 
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8.2.3 “Middle” Data Network Interface Restrictions 

Because of design limitations, it is not possible to receive messages via the 
“middle” Data Network interface. This is a permanent restriction — the corre¬ 
sponding recv registers in fact do not exist in the 2.2 version of the NI. 


8.2.4 Make Sure Doubleword Data Is Doubleword-Aligned 

C Programmers: This is also mentioned in the performance section above, but 
it doesn’t hurt to re-emphasize it. When you use doubleword read and write op¬ 
erations in your C code, you must compile your code with the -dalign compiler 
switch, so that doubleword values are properly aligned in memory: 

cc -c -g -DCM5 -dalign -I/usr/include ni_code.c 

If the doubleword values in your code are not properly aligned, the nodes will 
most likely signal “illegal address” errors, and your code won’t run. 


8.2.5 Order Is Important in Combine Messages 

As noted in Section 4.2.8, for scan messages longer than one word, the order in 
which the words of the message are written depends on the combine operation: 

* Maximum operations require the most si gnifi cant word to be written first. 

■ Both types of addition require the least significant word to be written first. 

■ Inclusive and exclusive OR have no word-ordering requirement. 


8.2.6 Broadcast and Combine Interface Conflicts 

Because of the way the broadcast and combine interfaces interact, you should be 
careful in using the abstain flags of these interfaces. If your code causes a node 
(processing node or PM) to abstain from the combine interface, and if 

■ the abstaining node is sending a broadcast message 

■ simultaneously, the other nodes are sending a combine message 
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then because of timing conflicts in the Control Network hardware, the two types 
of messages can collide, possibly causing your partition to crash. This situation 
most often occurs when you have instructed the PM to abstain from the combine 
interface so that it can receive the results of a scan or reduction operation, yet at 
the same time you want the PM to broadcast messages to the nodes telling them 
what to do. The conflict arises when the PM needs to broadcast a message at the 
same time that the nodes are sending a combine message. To avoid this problem, 
your code must include safety checks that prevent broadcast messages from 
backing up in the network at the same time that other nodes are sending a com¬ 
bine message. The CMost operating system includes a function you can call to 
send a broadcast message that implicitly performs this safety checking: 

int *msg, length; 

CMNA_bc_send_msg(msg, length); 


8.2.7 Broadcast Enabling 

As noted in Section 4.1.8, each broadcast interface has a send_enable flag. 
These flags are set to 0 by default in the CMOST operating system, and must be 
set to 1 before broadcasts are used. The CMOST system call to set these flags is: 

CMNA_participate_in(NI_BC_SEND_ENABLE); 

CMNA_par ticipate_in (NI_SBC_SEND_ENABLE) ; 


8.2.8 Combine Interface Pipelining Restriction 

As noted in Section 8.1.2, pipelined combine operations cannot be started using 
doubleword operations. However, you can use doubleword reads with pipelined 
operations, and doubleword writes are permitted for non-pipelined combine op¬ 
erations. 


8.2.9 Restriction on Scan Segment Start Flag 

As noted in Section 4.2.8, it is an error to change the state of the 
ni_scan_stazt register while the combine send FIFO is not empty. 
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8.2.10 Be Careful When Altering Abstain Flags 

As mentioned in Section 2.6.5, some programming systems use the abstain flags 
for their own purposes. When you alter the abstain flags, you must save the origi¬ 
nal settings and restore them before handing control back to these systems. 
Failing to do so can cause user or OS code to signal errors that are hard to trace. 


8.2.11 Simulating Receipt of Messages 

As noted in Section 3.4.3, a hardware defect in the NI chip does not allow recv 
registers to be written by the supervisor. The workaround is for a node to send 
a message into the network using its own address as the destination. Assuming 
the network is clear (as it is, for example, during context switches) this causes 
the message to be delivered to the front of the node’s receive queue. 


8.2.12 Message Too Long interrupt Restriction 

Currently, the message too long interrupt, described in Section B.3.7, does 
not work properly. The bus error still occurs, however. There is at present no 
workaround for this restriction. 


8.2.13 All Fall Down Restriction 

All Fall Down messages sometimes don’t set the all_£all_down bit in the 
private register The workaround for this restriction is to check that 
rec_JLenJLef t is greater than rec__len. 


8.2.14 Send/Receive and FIFO Locking Restrictions 

It is an error to send a message while in the middle of receiving one — a bug in 
the NI causes the receive status information to get written to the send status regis¬ 
ter. Also, it is an error to lock the send/receive FIFOs while in the middle of 
receiving a message — doing so can cause the remainder of the message to be 
lost. In general, it is best to lock and unlock the FIFOs only when both the send 
and receive FIFOs are clear. 
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Appendix A 

NI Registers, Fields, and Constants 


This appendix lists the registers and fields of the NI chip, as well as the constants 
used to locate them. To use these constants, either include the header file cmna. h 
(see Section 1.3.4), or the appropriate CMNA header file (see Appendix H). 

Note: The notation “2.2” indicates new registers/fields in Version 2.2 of the NI. 


A.1 NI Registers 

For each register the following information is provided: 

■ the name of the register 

■ the hex offset of the register from the user or supervisor base address 

■ the size of the register in bits, and its memory length in words 

■ the read/write permissions of the register for both user and supervisor 


Register Constants 

Note: With the exception of the send_£ irst and send_f irst_long registers 
(which are described in Section A.3 and A.4 below), the names of the constants 
used to access NI registers are derived from the names of the registers themselves 
by uppercasing the register name and adding the suffix “_a”. Each register 
constant provides the absolute address of the register, in either the user or super¬ 
visor memory area, depending on which header file (cmna. h or cmna_sup. h) 
has been included. 
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A.1.1 Global and System Registers 


Register Name: 

Address: 

Size: 

Len: 

Permissions: 
Super: User: 

ni_interrupt_cause 

0x0000 

15 

1 

R/W 

None 

ni_interrupt_cauBe_green 

0x0008 

14 

1 

R/W 

None 

ni_inte r rup t_leve1 

0x0010 

32 

1 

R/W 

None 

ni_physieal_self 

0x0018 

20 

1 

R/W 

None 

nl partition base 

0x0020 

20 

1 

R/W 

None 

nl partition size 

0x0028 

20 

1 

R/W 

None 

ni_chunk_taJDle_addr e s s 

0x0030 

6 

1 

R/W 

None 

ni_chunk_table_data 

0x0038 

8 

1 

R/W 

None 

ni_chunk_size 

0x0040 

3 

1 

R/W 

None 

ni_dr_message_count 

0x0048 

32 

1 

R/W 

None 

nl_count_mask 

0x0050 

16 

1 

R/W 

None 

ni_rec_interrupt_mask 

0x0058 

16 

1 

R/W 

None 

ni_u s e r_t ag_ma s k 

0x0060 

16 

1 

R/W 

None 

ni_time 

0x0070 

32 

1 

R/W 

R 

n±_eonfiguration 

0x0078 

5 

1 

R/W 

None 

ni_±nterrupt_send 

0x0080 

5 

1 

R/W 

None 

ni_serial_number 

0x0088 

32 

1 

R 

None 

nlavncCTlobal 

0x0090 

2 

1 

R 

R 

ni_sync_global_abs tain 

0x0098 

1 

1 

R/W 

R/W 

ni_eom_flush_send 

OxOOAO 

1 

1 

W 

None 

ni_async_global 

Qx00A8 

2 

1 

R/W 

R/W 

ni_async_sup_global 

OxOOBO 

2 

1 

R/W 

None 

ni_hodgepodge 

0x00B8 

6 

1 

R/W 

None 

ni_sync_global_send 

OxOOCO 

1 

1 

R/W 

R/W 

ni_interrupt_clear 

0x00C8 

15 

1 

W 

None 

ni_interrupt_clear_green 

OxOODO 

14 

1 

w 

None 

ni_interrupt_now 

0x00D8 

32 

1 

R/W 

None 

ni_scaja_start 

OxOOEO 

1 

1 

R/W 

R/W 

ni_bad_addxes6 

0x00E8 

32 

1 

R/W 

None 

ni_l onge s t_dr_me s a age 

0x0160 

5 

1 

R/W 

R 2.2 

ni_uaer_rec_in.terrupt_mask0x0168 

16 

1 

R/W 

R/W 22 

ni_interrupt_set 

0x0190 

20 

1 

W 

None 2.2 

ni_interxupt_set__green 

0x0198 

19 

1 

w 

None 22 
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A.1.2 Network Interface Registers 

Combined Data Network interface (DR) 


Permissions: 


Register Name: 

Address: 

Size: 

Len: 

Super: 

User: 


ni_dr_s tatus 

0x0200 

24 

1 

R/W 

R 


nl dr orivate 

0x0208 

10 

1 

R/W 

None 


ni_dx_send 

0x0230 

32 

16 

W 

W 


nl_dx_status_all 

0x0250 

32 

1 

R 

R 

2.2 

nl_dr_s tatus_long 

0x0260 

32 

1 

R 

R 

2.2 

nl_dr_send_f irst (block) 

0x1000 

32 

2 

W 

W 


ni_dr_send_f lrst_long 

0x8000 

32 

2 

W 

W 

22 


Left Data Network Interface (LDR) 


Permissions: 


Register Name: 

Address: 

Size: 

Len: 

Super: 

User: 


ni_ldr_s tatus 

OxOcOO 

32 

1 

R/W 

R 


nl ldr private 

0x0c08 

24 

1 

R/W 

None 


n i_l dx_r e cv 

0x0c20 

32 

16 

R/W 

R 


ni_ldr_send 

0x0c30 

32 

16 

W 

W 


nl ldr status pop 

0x0c40 

32 

2 

R 

R 

2.2 

ni__ldx_status_all 

0x0c50 

32 

1 

R 

R 

22 

nl_ldr_status_long 

0x0c60 

32 

1 

R 

R 

22 

ni_ldr_send_f Irst (block) 0x6000 

32 

2 

W 

W 


ni_ldx__send_fixst_long 0x10000 

32 

2 

W 

W 

2.2 


Right Data Network interface (RDR) 


Permissions: 


Register Name: 

Address: 

Size: 

Len: 

Super: 

User: 


nl_rdr_s tatus 

OxOeOO 

32 

1 

R/W 

R 


ni rdr private 

0x0e08 

24 

1 

R/W 

None 


ni_rdr_recv 

0x0e20 

32 

16 

R/W 

R 


nl_rdr_send 

0x0e30 

32 

16 

W 

W 


ni_rdr_s tatus pop 

0x0e40 

32 

2 

R 

R 

2.2 

ni_rdr_s tatus_al1 

0x0e50 

32 

1 

R 

R 

2.2 

ni_r dr_s t a tus_l ong 

0x0e60 

32 

1 

R 

R 

22 

ni_xdr_send_£ixst (block)0x7000 

32 

2 

W 

W 


ni_rdx_send_f ixst_long 

0x18000 

32 

2 

W 

W 

2.2 


NI Version 2.2 (CM-5E), June 1994 147 

Copyright © 1994 Thinking Machines Corporation 





NIProgrammer’s Handbook 


Broadcast Interface (BC) 


Permissions: 


Register Name: 

Address: 

Size: 

Len: 

Super: 

User: 

ni_bc_s tatus 

0x0600 

6 

1 

R 

R 

nibeprivate 

0x0608 

17 

1 

R/W 

None 

ni_bc__contxol 

0x0610 

1 

1 

R/W 

R/W 

ai_bc_recv 

0x0620 

32 

16 

R/W 

R 

ni__bc_send 

0x0630 

32 

16 

W 

W 

ni_bc_send_first (block) 

0x3000 

32 

2 

W 

W 

Supervisor Broadcast Interface (SBC) 








Permissions: 

Register Name: 

Address: 

Size: 

Len: 

Super: 

User: 

ni_sbc_s tatus 

0x0800 

6 

1 

R 

None 

nisbc private 

0x0808 

17 

1 

R/W 

None 

ni_sbc_control 

0x0810 

1 

1 

R/W 

None 

ni_abc_recv 

0x0820 

32 

16 

R/W 

None 

ni_abc_send 

0x0830 

32 

16 

W 

None 

n.i_sbc_aend_f irst (block) 0x4000 

32 

2 

W 

None 

Combine Interface (COM) 









Permissions: 

Register Name: 

Address: 

Size: 

Len: 

Super: 

User: 

ni_com_atatus 

OxOaOO 

12 

1 

R/W 

R 

ni_oom_private 

0x0a08 

6(18) 

1 

R/W 

None 

ni_coin_con.trol 

OxOalO 

2 

1 

R/W 

R/W 

ni_com_recv 

0x0a20 

32 

16 

R/W 

R 

ni_com_send 

0x0a30 

32 

16 

R/W 

W 

ni_com_send_f irst (block) 0x5000 

32 

2 

W 

W 
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A.2 NI Message Length Limit Constants 

The following constants give the message length limits of the network interfaces: 

MAX_ROUTER_MSG_WORDS DR/LDR/RDR interface length limit. 

max_combine_msg_words Combine (COM) interface length limit. 

MAX_B road CAS t_ms g_word s Broadcast (BC) interface length limit. 
MAX_SBC_MSG_woRDS Supervisor broadcast (SBC) length limit. 

These constants determine the maximum values that can be supplied in the length 
component of the auxiliary data of a network message. (See the descriptions of 
the auxiliary data formats for the various interfaces below.) 


A.3 Send First Register Addresses 

The send_f irst address for a network message is a 32-bit value of the form 


31 

□ 


15 14 


12 11 


0 


base address 


interface 


auxiliary data 


0 0 0 


where interface is the interface number (an integer from 0 to 7 representing the 
interface being used), and auxiliary data is the auxiliary information of the mes¬ 
sage. (The base address portion is the base address of the NI memory area, either 
user or supervisor.) 

The following constants are used to construct send_f irst addresses: 


NI_BASE 
SF_FIFO_OFFSET 
AUXILIARY START P 


The NI base address. 

The interface field offset (12). 

The auxiliary data field offset (3). 


To construct a send_f irst address, add the following values, left-shifted as 
shown: 


The NI base address: 
The interface constant: 
The auxiliary data: 


NI_BASE 

interface_number « SF_FlFO_OFFSET 
auxiliary,_data « auxiliary_START_p 


NI Version 2.2 (CM-5E), June 1994 

Copyright © 1994 Thinking Machines Corporation 


149 




NI Programmer’s Handbook 


The following interface jmmber constants are defined: 


data_router_fifo DR network interface (1). 

left_dr_fifo LDR network interface (6). 

rxght_dr_fxfo RDR network interface (7). 

user_bc_fxfo User broadcast (BC) interface (3). 

supervi sor_bc_fifo Supervisor broadcast (SBC) interface (4). 
combine_fifo Combine (COM) interface (5). 

The constants specifying the auxiliary data format for each interface are listed 
in the sections below. 


Data Network (DR/LDR/RDR) Auxiliary Data Fields 


The format of the auxiliary data of a Data Network message is 
8 4 0 


md\ 


tag 


length 


where 


md is the addressing mode (0 = relative, 1 *= physical). 
tag is the 4-bit tag value. 

length is the length of the message in words, excluding address word. 


The following constants specify the starting bit positions of these fields: 


NI_DR_SEND_AtJXIL IARY_ADDRES S_MODE_P The md field offset (8) . 
ni_dr_send_auxiliary_tag_p The tag field offset (4). 

NI_DR_SEND_AUXILIARY_LENGTH_P The length field offset (0). 

To construct a send_f irst address, add the following values: 

The md flag: md « ni_dr_send_auxi l iary_addres s_mode_p 

The tag value: tag « ni_dr_send_auxilxary_tag_p 

The length value: length « ni_dr_s END_AUXX LI ary_length_p 

The following constants can be used to specify the md flag: 

relative Relative node addressing (0). 

physical Physical node addressing (1). 

The tag can be any value from 0 to 3 inclusive for user messages, or from 0 to 
15 for supervisor messages. (The length value limit is given in Section A.2.) 
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Broadcast (BC/SBC) Auxiliary Data Fields 


The format of the auxiliary data of a broadcast message is: 


8 


0 


0 0 0 0 0 


length 


where length is the length of the message in words. (The high-order bits of the 
auxiliary data have no useful meaning, but must always be 0.) The following 
constant specifies the starting bit position of the length field: 

Nl_BC_SENT>_AUXlLlARy_LENGTH_i> The length field offset (0). 


Combine Auxiliary Data Fields 

The format of the auxiliary data of a combine interface message is: 


where 


8 


0 


pattern 


combiner 


length 


pattern is a two-bit value selecting the order in which values are combined 
combiner is a three-bit value selecting the combine operation performed 
length is the length of the message in words 

The following constants specify the starting bit positions of these fields: 

NI_COM_S END__Atrxi LI ARY_PATTERN_F The pattern field offset (7). 
ni_com_sEND_AUXILIary_combiner_p The combiner field offset (4). 
NI_COM_SEND_AUXILIARY_I»ENGTH_P The length field offset (0). 

To construct a send_f irst address, add the following values: 

The pattern value: pattern « ni_com_seito_auxiliary_pattekn_p 
The combiner value: combiner « Nl_COM_SEND_AUXILIARY_COMBINER_P 
The length value: length « ni_com_send_auxiliary_length_p 

The foEowing constants can be used to specify the value of the pattern field: 

s can_forwakd Forward scan pattern (2). 

S can_backward Backward scan pattern (1). 
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scan_reduce Reduction scan pattern (3). 

SCAN_router_done Network-done operation (0). 


The following constants can be used to specify the value of the combiner field: 


OR_SCAN 

ADD_SCAN 

XOR_SCAN 

UADD_SCAN 

MAX_SCAN 

ASSERT ROUTER DONE 


Inclusive OR (0). 

Signed addition (1). 
Exclusive OR (2). 

Unsigned add (3). 

Signed maximum (4). 
Network-done operation (5). 


A.4 Send First Long (Data Network) Register Addresses 


The send_f irst_long address for a Data Network message is a 32-bit value 
of the form 


31 

□ 


19 


17 


15 


13 12 


base addr 


0 0 


intf 


0 0 


0 


auxiliary data 


0 0 0 


where intf is the interface number (an integer from 0 to 3 representing the Data 
Network interface being used), and auxiliary data is the auxiliary information. 
(The base address portion is the base address of the NI memory area, user or 
supervisor.) 

The following intf values are defined: 


0 - Not used 
1 - DR network interface 


2 - LDR network interface 

3 - RDR network interface 


The format of the auxiliary information is: 

9 4 


where 


0 


md 


length 


tag 


md is the addressing mode (0 = relative, 1 = physical). 

length is the length of the message in words, excluding address word. 

tag is the 4-bit tag value. 
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A.5 NI Fields 

The register subfields of the NI are presented below, grouped by register. For 
each field, the following information is provided: 

■ the name of the field 

■ the name of the position constant used to access the field (see note below) 

■ the starting position and bit length of the field 

■ the read/write permissions of the field for both user and supervisor 

Note: The progr amming constants used to access NI fields come in pairs. 

One constant, with a suffix of “_P”, gives the starting bit position of the field. 
In the tables below, this value appears in the Pos: (position) column. 

The other constant, with a suffix of gives the length of the field. In the 
tables below, this value appears in the Len: (length) column. 

Only the “_j?” constant name is shown in the tables below. Unless otherwise 
noted, you can assume that the “_L” constant exists as well. 


A.5.1 Combined Data Network (DR) Fields 
The ni_dr_status Register 


Permissions: 


Field Name: 

Constant: 

Pos: 

Len: 

Super: 

User: 

ni send space . 

NX SEND SPACE P . 

. 0 

4 

R 

R 

ni rec ok . 

NI SEC OK P . 

. 4 

1 

R 

R 

ni send ok . 

NX SEND OK P . 

. 5 

1 

R 

R 

ni_r outer_done_c omp 1 e t e 

NI_ROtJTER_DONE_COMPLETE_P 

. 6 

1 

R 

R 

ni rec length left . 

NI_RECJC,ENGTH_LEFT_P . . 

. 7 

4 

R/W 

R 

ni rec length . 

NX SEC LENGTH P . 

. 11 

4 

R/W 

R 

ni dr rec tag . 

NX DR REC TAG P . 

. 15 

4 

R/W 

R 

ni dr send state . 

NX DR SEND STATE P . . . . 

.21 

2 

R 

R 

ni dr rec state . 

NX DR REC STATE P . 

.23 

2 

R 

R 
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The ni_dr_statusJong Register 


Permissions: 


Field Name: 

Constant: Pos: 

Len: 

Super: 

User: 


ni_send_space . 

NI_SEND_SPACE_LONG_P ... 0 

5 

R 

R 

2.2 

ni ree ok . 

NI EEC OK LONG P . 5 

1 

R 

R 

2.2 

ni send ok . 

NX SEND OK LONG P.6 

1 

R 

R 

2.1 

ni_router_done_complete 

KI i _ROOTER_I>ONE_COHPIiSTE__IfOKa_P ... 7 

1 

R 

R 

2.2 

ni rec length left . 

NI_REC_LENGTH_LEFT_LONG_P 8 

5 

R/W 

R 

2.2 

ni rec length . 

NI REC LENGTH LONG P ... 13 

5 

R/W 

R 

2.2 

ni_dr_rec_tag .. 

NI_DR_REC_TAG_LONG_P ... 18 

4 

R/W 

R 

22 

ni dr send state . 

NX DR SEND STATE LONG P 24 

2 

R 

R 

22 

ni dr rec state . 

NI_DR_REC_STATE_LONG_P . 26 

2 

R 

R 

22 

The ni_dr_status_{al!/pop} Registers 








Permissions: 


Field Name: 

Constant: Pos: 

Len: 

Super: 

User: 


ni ldr rec ok .. 

. 0 

1 

R 

R 

2.2 

ni rdr rec ok . 

. 1 

1 

R 

R 

22 

ni dr send ok . 

. 2 

1 

R 

R 

22 

ni ldr rec tag . 

. 3 

4 

R 

R 

22 

ni rdr rec tag . 

. 7 

4 

R 

R 

22 

ni_ldr_rec_length_long . 

. 11 

5 

R 

R 

22 

ni_rdr_rec_length_long . 

. 16 

5 

R 

R 

22 

ni dr send space . 

.21 

5 

R 

R 

22 

ni ldr rec all fall down.26 

1 

R 

R 

22 

ni rdr rec all fall down.27 

1 

R 

R 

22 

ni_router_done_complete 

.31 

1 

R 

R 

22 


The ni_dr_private Register 


Permissions: 

Field Name:_Constant:_Pos: Len: Super: User: 


ni_rec_pk_ie . 

ni_lock . 

ni_rec_stop . 

ni__rec_full . 

ni_dr_rec_al l_f al l_down 
ni__all_f all_down_ie . . . . 
ni_all_fall_dovm_enable 
ni_sfif o_goes_empty_ie . 
ni_rdone__conrplete_ie . . . 


NI_REC_OK_IE_P . 

NI_LOCK_P . 

NX_REC_STOP_P . 

NI__REC_FULL_P. 

NI_DR_REC_ALL_FALL_DOWN_P . 
NI_ALL_FALL_DOWN_IE_P . . . . 
NI_*LL_FALL_DOWN_ENABLE_P 
NI_SFIFO_GOES_EMPTY_IE__P . 
NX RDONE COMPLETE IE P . . . 


0 

1 

R/W 

None 

1 

1 

R/W 

None 

2 

1 

R/W 

None 

3 

1 

R 

None 

5 

1 

R/W 

None 

6 

1 

R/W 

None 

7 

1 

R/W 

None 

8 

1 

R/W 

None 

9 

1 

R/W 

None 


12 

12 
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A.5.2 Left Data Network Interface (LDR) Fields 
The ni_ldr_status Register 


Permissions: 


Field Name: 

Constant: 

Pos: 

Len: 

Super: 

User: 

ni send space . 

... in SEND SPACE P . 

. 0 

4 

R 

R 

ni rec ok . 

... NX REC OK P . 

. 4 

1 

R 

R 

ni send ok . 

... NX SEND OK P . 

. 5 

1 

R 

R 

ni_rec_length_le£ t. 

. . . NI_REC_LENGTH_LEFT_P . 

. 7 

4 

R/W 

R 

ni rec length . 

- - - NI REC LENGTH P . 

. 11 

4 

R/W 

R 

ni dr rec tag . 

. . . NI DR REC TAG P . 

. 15 

4 

R/W 

R 

The nijdr_statusJong Register 









Permissions: 

Field Name: 

Constant: 

Pos: 

Len: 

Super: 

User: 


ni send space . 

. NI_SEND_S PACE_LONG_P 

... 0 

5 

R 

R 

2.2 

ni rec ok . 

. NI_REC_OK_LONG_P . . . . 

... 5 

1 

R 

R 

2.2 

ni send ok . 

. NI_SEND_OK_LONG_P . . . 

... 6 

1 

R 

R 

2.2 

ni_rec_l@ngth_le£t.... 

. NIJTCCJLENGTH_I^FT__LONG_J? 

... 8 

5 

R/W 

R 

2.2 

ni rec length . 

. NX REC LENGTH LONG P 

... 13 

5 

R/W 

R 

22 

ni dr rec tag . 

. NI_DR_REC_TAG_LONG_P 

...18 

4 

R/W 

R 

2.2 


The niJdr_status_{all/pop} Registers 

Permissions: 

Field Name:_Constant:_Pos: Len: Super: User: 


ni ldr rec ok . 

. 0 

1 

R 

R 

2.2 

ni rdr rec ok . 

. 1 

1 

R 

R 

22 

ni ldr send ok . 

. 2 

1 

R 

R 

22 

ni ldr rec tag . 

. 3 

4 

R 

R 

22 

ni rdr rec tag . 

. 7 

4 

R 

R 

2.2 

ni ldr rec length long . 

. 11 

5 

R 

R 

22 

ni rdr rec length long . 

. 16 

5 

R 

R 

22 

ni ldr send space . 

. 21 

5 

R 

R 

22 

ni ldr rec all £all down . 

.26 

1 

R 

R 

22 

ni rdr rec all £all down . 

. 27 

1 

R 

R 

2.2 

ni router done complete . 

.31 

1 

R 

R 

22 
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The ni_ldr_private Register 


A.5.3 


Permissions: 


Field Name: 

Constant: 

Pos: 

Len: 

Super: 

User: 

ni rec ok ie . 

, NI REC OK IE P . 

. 0 

1 

R/W 

None 

ni lock . 

, NI LOCK P . 

. 1 

1 

R/W 

None 

nl rec full . 

. NI REC FULL P . 

. 3 

1 

R 

None 

ni_dr_rec_all_fall_do»m . . 

. NI_DR_REC_ALL_FALL_DOWH_P 

. 5 

1 

R/W 

None 


Right Data Network interface (RDR) Fields 

The ni_rdr_status Register 

Field Name: Constant: 

Pos: 

Len: 

Permissions: 
Super: User: 

nl send space . 

. . NI SEND SPACE P. 

. 0 

4 

R 

R 

ni zee ok . 

. . NI REC OK P. 

. 4 

1 

R 

R 

nl send ok . 

. . NI SEND OK P. 

. 5 

1 

R 

R 

ni_rec_length_left. . . 

. . NI_REC_LENGTH__LEFT_P . . 

. 7 

4 

R/W 

R 

nl zee length . 

. . NI REC LENGTH P. 

. 11 

4 

R/W 

R 

ni dr zee tag . 

. . NI DR REC TAG P. 

. 15 

4 

R/W 

R 


The ni_rdr_status_long Register 

Field Name: Constant: 

Pos: 

Len: 

Permissions: 
Super: User: 


ni send space . 

NI_SEND_SPACE_LONG_P 

... 0 

5 

R 

R 

2.2 

ni rec ok . 

NI REC OK LONG P . . . . 

.. . 5 

1 

R 

R 

2.2 

ni send ok . 

NI_SEND_OK_LONG_P . . . 

... 6 

1 

R 

R 

2.2 

ni zee length left .... 

KI__REC_LENGTH_LErT__LONO_P 

... 8 

5 

R/W 

R 

22 

ni rec length . 

NI REC LENGTH LONG P 

...13 

5 

R/W 

R 

22 

ni dr zee tag . 

NI_DR__REC_TAG_LONG_P 

...18 

4 

R/W 

R 

22 
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The ni_rdr_status_{all/pop} Registers 

Permissions: 


Field Name: Constant: 

Pos: 

Len: 

Super: 

User: 


ni rdr rec ok . 

. 0 

1 

R 

R 

2.2 

ni ldr rec ok . 

. 1 

1 

R 

R 

2.2 

ni rdr send ok . 

. 2 

1 

R 

R 

2.2 

nl rdr rec tag . 

. 3 

4 

R 

R 

2.2 

ni ldr rec tag . 

. 7 

4 

R 

R 

2.2 

ni rdr rec length long.. 

. 11 

5 

R 

R 

2.2 

ni ldr tec length long . 

. 16 

5 

R 

R 

2.2 

ni rdr send space . 

. 21 

5 

R 

R 

2.2 

ni rdr rec all fall down . 

.26 

1 

R 

R 

2.2 

ni ldr rec all fall down . 

. 27 

1 

R 

R 

2.2 

ni router done complete . 

. 31 

1 

R 

R 

2.2 


The ni_rdr_private Register 


Field Name: 

Constant: 

Pos: 

Len: 

Permissions: 
Super: User: 

ni rec ok ie . 

. NX REC OK IE P . 

. 0 

1 

R/W 

None 

ni lock . 

. NI LOCK P . 

. 1 

1 

R/W 

None 

ni rec full . 

. NI REC FULL P . 

. 3 

1 

R 

None 

n i_dr_r e c_al 1 _£ a 1 l_do wn . . 

, NI_DR_RBC_ALL_FALL_DOWN_P 

. 5 

1 

R/W 

None 


A.5.4 Broadcast interface (BC) Fields 
The ni_bc_status Register 

Permissions: 

Field Name: _ Constant: _ Pos: Len: Super: User: 

ni_send_space .NI_SEND_SP ACE_P. 0 4 R R 

ni_rec_ok ...NI_REC_OK_P.4 1 R R 

ni_send_ok .Nl_SEND_OK_P. 5 1 R R 

ni_send_enrpty .NI_SEND_EMPTY_P.6 1 R R 

ni_rec_length_left .NI_REC_LENGTH_LEFT_P ... 7 7 R R 2.2 

(si_BC_RKC_LENGTH_LErr_LONG_L) 
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The ni_bc_private Register 


A.5.5 


Permissions: 


Field Name: 

Constant: 

Pos: 

Len: 

Super: 

User: 

ni rec ok ie . 

_ NX REC OK XE P . 

.... 0 

1 

R/W 

None 

ni lock . 

_ NI LOCK P . 

.... 1 

1 

R/W 

None 

ni zee stop . 

.... NX REC STOP P . 

.... 2 

1 

R/W 

None 

ni rec full . 

.... NX REC POLL P . 

.... 3 

1 

R 

None 

ni_send_enable .... 

.... NI_SEND_ENABLE_P . . . 

.... 4 

1 

R/W 

None 


The ni_bc_control Register 




Field Name: 

Constant: 

Pos: Len: 

Permissions: 
Super: User: 

ni_rec_abstain .... 

_NI_REC_ABSTAIN_P . . . 

.... 0 1 

R/W 

R/W 


Supervisor Broadcast Interface (SBC) Fields 

The ni_sbc_status Register 

Field Name: Constant: 

Pos: 

Len: 

Permissions: 
Super: User: 

ni_send_space . 

_ NX SEND SPACE P . 

0 

4 

R 

None 

ni rec ok . 

_ NI REC OK P . 

4 

1 

R 

None 

ni send ok . 

_ NX SEND OK P . 

5 

1 

R 

None 

ni send empty . 

_ NI SEND EMPTY P . 

6 

1 

R 

None 

ni_rec_length_lef t ., 

. . . . NI_REC_LKNGTH_LEFT_P. 

7 

4 

R 

. None 


The ni_sbc_private Register 

Field Name: Constant: 

Pos: 

Len: 

Permissions: 
Super: User: 

ni_rec_ok_ie .. 

.NX REC OK IE P.. 

.... 0 

1 

R/W 

None 

ni lock . 

.NI LOCK P.. 

.... 1 

1 

R/W 

None 

ni_send_stop .. 

. NX SEND STOP P . 

.... 2 

1 

R/W 

None 

ni rec full ... 


.... 3 

1 

R 

None 

ni_send_enable 


.... 4 

1 

R/W 

None 
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ni_rec_abstain . NI_REC_ABSTAIN_P . 0 1 R/W None 


A.5.6 Combine interface (COM) Fields 
The ni_com_status Register 


Permissions: 


Field Name: 

Constant: 

Pos: 

Len: 

Super: 

User: 

ni send space . 

NX SEND SPACE P. 

. 0 

4 

R 

R 

ni rec ok . 

NI REC OK P . 

. 4 

1 

R 

R 

ni send ok . 

NX SEND OK P. 

. 5 

1 

R 

R 

ni send empty . 

NX SEND EMPTY P . 

. 6 

1 

R 

R 

ni rec length left . 

NI_REC_LENGTH_LEFT_P . . 

. 7 

4 

R/W 

R 

ni rec length . 

NX REC LENGTH P . 

. 11 

4 

R/W 

R 

ni_c om_s c an_ove r f1ow... 

NX_COM_SCAN_OVERPLOW_P 

.20 

1 

R/W 

R 


The ni_com_private Register 


Reid Name: 

Constant: 

Pos: 

Len: 

Permissions: 
Super: User: 

ni rec ok ie . 

NX REC OK IE P . 

0 

1 

R/W 

None 

ni lock . 

NI LOCK P . 

1 

1 

R/W 

None 

ni rec stop . 

NX REC STOP P . 

2 

1 

R/W 

None 

ni rec full . 

NI REC FTJLL P. 

3 

1 

R 

None 

ni_com_scan_overflow_ie 

NI_COM_SCAN_OVERFLOW_IE_P . 

4 

1 

R/W 

None 

ni_c om_r e c_emp ty_ie .... 

NX_CQM_REC_EMPTY_IE_P . . 

5 

1 

R/W 

None 

ni com send length. 

NI_COM_S END_LENGTH_P . . . 

8 

4 

R 

None 

ni_com_send_combiner ... 

NI_CQM_SEND_COMBINER_P . 

12 

3 

R 

None 

ni com send pattern.... 

NI_COM_SEND_PATTERN_P . . 

15 

2 

R 

None 

ni com send start . 

NI_COM_SEND_STARTJP_ 

17 

1 

R 

None 
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The nLcom_controI Register 

Permissions: 

Field Name: _ Constant: _ Pos: Len: Super: User: 

ni_rec_abstain . NI_R£C_ABSTAXNJP . 0 1 R/W R/W 

ni_xeduce_rec_abstain. . NIJRED0CE_REC_ABSTAXN_P 1 1 R/W R/W 


A.5.7 Global Interface Fields 

The ni_sync_gIobal Register 

Permissions: 

Field Name: _ Constant: _ Pos: Len: Super: User: 

ni_sync_global_rec.NI_SYNC_GX.OBALJREC_P ... 0 1 R R 

ni_sync_global_complete ni_s«jc_global_complete_p .11 R R 


The ni_async_globai Register 

Permissions: 

Field Name: _ Constant: _ Pos: Len: Super: User: 

ni_global_send . NI_GLOBAL_SEND_P . 0 1 R/W R/W 

ni_global_rec . NX_GLOBAL_REC_P . 1 1 R R 


The ni_async_sup_globai Register 


Reid Name: 


Constant: 


Permissions: 
Pos: Len: Super: User: 


ni_supervisor_global_aeud HX_SUPEEVISOR_GI.OBAL_SEND_P 0 1 

al_Bupervisor_global_rec . NX_SOTERvrsoR_GLOBAL_REC_P 1 1 


R/W None 
R None 


A.5.8 Interrupt Register Fields 

Note: The position (“_p”) constants for these flags are as described above. The 
length for all flags (1) is given by the single constant ni_xnterrupt_l . To lo¬ 
cate the flags in the interrupt_clear/set registers, use the constants 
defined for the interrupt_cause registers — the flag positions are the same. 
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The ni_interrupt_cause Register 


Flag Name: 

Pos: 

Len: 

Permissions: 
Super: User: 

ni cause internal £ault . 

... 0 

1 

R/W 

None 

ni cause me error . 

... 1 

1 

R/W 

None 

ni cause emu error . 

... 2 

1 

R/W 

None 

ni cause be interrupt red . 

... 3 

1 

R/W 

None 

ni cause cn oh«rik«tim etrrar . 

. .. 4 

1 

R/W 

None 

ni_cause_cn_hard_error . 

... 5 

1 

R/W 

None 

ni cause dr checksum error . 

... 6 

1 

R/W 

None 

ni cause timer interrupt . 

... 7 

1 

R/W 

None 

ni cause be interrupt orange . 

... 8 

1 

R/W 

None 

ni cause be interrupt_yellow . 

... 9 

1 

R/W 

None 

ni cause be or com collision . 

... 10 

1 

R/W 

None 

ni_cause_com_abstain_changed. 

... 11 

1 

R/W 

None 

ni cause dr count negative . 

... 12 

1 

R/W 

None 

ni_cause_bad_relative_address . . .. 

... 13 

1 

R/W 

None 

ni cause bad memory access . 

... 14 

1 

R/W 

None 

ni cause message too long . 

... 15 

1 

R/W 

None 2.2 

ni cause rdone complete . 

... 16 

1 

R/W 

None 2.2 


The nijnterrupt_cause_green Register 


Flag Name: 

Pos: 

Len: 

Permissions: 
Super: User: 

ni cause be interrupt_green . 

... 0 

1 

R/W 

None 

ni eaime scan mmrflow. 

... 1 

1 

R/W 

None 

ni cause be rec ok. 

... 2 

1 

R/W 

None 

ni cause sbe rec ok . 

... 3 

1 

R/W 

None 

ni cause com rec ok . 

... 4 

1 

R/W 

None 

ni cause com rec empty . 

... 5 

1 

R/W 

None 

ni cause svnc global rec . 

... 6 

1 

R/W 

None 

ni cause global rec . 

... 7 

1 

R/W 

None 

ni_cause_supervisor_global_rec . . . 

... 8 

1 

R/W 

None 

ni cause dr rec ok. 

... 9 

1 

R/W 

None 

ni cause ldr rec ok . 

... 10 

1 

R/W 

None 

ni cause rdr rec ok . 

... 11 

1 

R/W 

None 

ni cause dr rec tag . 

... 12 

1 

R/W 

None 

ni cause dr rec all fall down .... 

... 13 

1 

R/W 

None 

ni cause ldr rec tag . 

... 14 

1 

R/W 

None 2.2 

ni cause rdr rec tag . 

... 15 

1 

R/W 

None 22 

ni cause ldr user rec tag . 

. .. 16 

1 

R/W 

None 2.2 

ni cause rdr user rec tag . 

... 17 

1 

R/W 

None 2.2 

ni cause sfifo empty . 

... 18 

1 

R/W 

None 2.2 

ni cause dperr . 

... 19 

1 

R/W 

None 2.2 
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Permissions: 


Field Name: 

Pos: 

Len: 

Super: 

User: 

nl_ {clear,set}_intexnal_£ault. 

0 

1 

W 

None 

ni_{clear,set}_mc_error . 

1 

1 

W 

None 

ni_{clear,set}_cmu_error . 

2 

1 

W 

None 

ni {clear,set} be Interrupt red ..... 

3 

1 

W 

None 

ni.{clear,set}cnchecksum,.error .... 

4 

1 

W 

None 

ni {clear,set} cn hard error . 

5 

1 

W 

None 

ni_{clear,set}_dr_checksum_error .... 

6 

1 

W 

None 

ni {clear,set} timer interrupt . 

7 

1 

W 

None 

ni_{clear,set}_bc_interrupt_orange . . 

8 

1 

W 

None 

ni_{clear,set}_bc_interrupt_yellow . . 

9 

1 

W 

None 

ni_{clear,set}_bc_or_com_collision . . 

10 

1 

W 

None 

ni_{clear,set}_com_abstain_changed . . 

11 

1 

W 

None 

ni_{clear,set}_dr_count_negative .... 

12 

1 

W 

None 

ni_{clear,set}__bad_relative_address . 

13 

1 

W 

None 

ni__ {clear, set}_bad_memory__access .... 

14 

1 

W 

None 

ni {clear,set} message too long . 

15 

1 

W 

None 2.2 

ni {clear,set} rdone complete. 16 

The ni_interrupt_{ciear,set}_green Registers 

1 

W None 2.2 

Permissions: 

Field Name: 

Pos: 

Len: 

Super: 

User: 

ni_{clear,set}_bc_interrupt_green .. . 

0 

1 

W 

None 

ni {clear,set} scan overflow. 

1 

1 

W 

None 

ni {clear,set} be rec ok. 

2 

1 

W 

None 

ni {clear,set} sbe rec ok. 

3 

1 

W 

None 

ni_{clear,set}_com_rec_ok . 

4 

1 

W 

None 

ni {clear,set} com rec empty . 

5 

1 

W 

None 

ni {clear,set} svnc global rec . 

6 

1 

W 

None 

ni {clear,set}_global rec . 

7 

1 

W 

None 

ni_ {clear,set}_supervisor_global_rec 

8 

1 

W 

None 

ni {clear,set} dr rec ok. 

9 

1 

W 

None 

ni {clear,set} Idr rec ok. 

10 

1 

W 

None 

ni {clear,set} rdr rec ok. 

11 

1 

W 

None 

ni {clear,set} dr rec tag. 

12 

1 

w 

None 

ni_{clear, set}_dr_rec_all_fall_dowa . 

13 

1 

w 

None 

ni {clear,set} ldr rec tag. 

14 

1 

w 

None 2J2 

ni {clear,set} rdr rec tag. 

15 

1 

w 

None 2J2 

ni {clear,set} ldr user rec tag . 

16 

1 

w 

None 22 

ni {clear,set} rdr user rec tag . 

17 

1 

w 

None 2.2 

ni {clear,set} sfifo empty . 

18 

1 

w 

None 22 

ni {clear,set} dperr. 

19 

1 

w 

None 2.2 
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A.5.9 Other Register Fields and Constants 

Note: The programming constants for these flags are obtained by uppercasing 
the name of the flag, then adding “_p” for the position, or “_l” for the length. 

The niJnterruptJevel Register 


Permissions: 

Field Name: Pos: Len: Super: User: 


ni interrupt level green . 

... 0 

1 

R/W 

None 

ni interrupt level_yellow. 

... 8 

1 

R/W 

None 

ni interrupt level orange . 

... 16 

1 

R/W 

None 

ni interrupt level red . 

... 24 

1 

R/W 

None 

The nijhodgepodge Register 








Permissions: 

Field Name: 

Pos: 

Len: 

Super: 

User: 


ni global rec ie . 

. . . . 0 

1 

R/W 

None 

ni_supervisor_global_rec_ie . 

. . . . 1 

1 

R/W 

None 

ni flush complete . 

.... 2 

1 

R 

None 

ni interrupt send ok . 

... . 3 

1 

R 

None 

ni configuration complete . 

. . . . 4 

1 

R 

None 

ni interrupt rec enable . 

. . .. 5 

1 

R/W 

None 

ni sync global rec ie . 

.... 6 

1 

R/W 

None 

ni timer ie . 

.... 7 

1 

R/W 

None 

ni cn stop send . 

.... 8 

1 

R/W 

None 

ni disable bus error . 

.... 9 

1 

R/W 

None 2.2 

ni ldr rec tag ie . 

.... 10 

1 

R/W 

None 2.2 

ni rdr rec tag ie . 

.... 11 

1 

R/W 

None 2.2 

ni ldr user rec tag ie .. 

.... 12 

1 

R/W 

None 2.2 

ni rdr user rec tag ie . 

.... 13 

1 

R/W 

None 2.2 

ni msg too long ie.. 

.... 14 

1 

R/W 

None 2.2 
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The ni_bad_address Register 





Permissions: 

Field Name: 

Pos: 

Len: 

Super: 

User: 

n± bad address low . 

. 0 

20 

R/W 

None 

ni bad address type . 

. 20 

12 

R/W 

None 


Note: The contents of the n±_bad_address register are implementation- 
dependent, so there are no predefined constants for t hi s register. 
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Appendix B 

NI Interrupts 


The methods used to recover from an NI interrupt depend heavily on the type of 
interrupt itself. This appendix describes each of the possible interrupts in detail, 
and provides guidelines for recovering from them. 

For each interrupt, the following information is provided: 

■ the name and color of the interrupt 

■ the ni_interrupt_cause or ni_lnteriupt_cause_green flag that 
is set when the interrupt is signaled 

■ the ni_interrupt_clear or ni_interrupt_clear_green. flag that 
is used to clear the interrupt when it has been handled 

■ the ni_interrupt_set or ni_interrupt_set_green flag that is 
used to artificially trigger the interrupt 

■ the triggering event that causes the interrupt to be signaled 

■ the effect of the interrupt on the NI and the networks 

■ the correct method for handling the interrupt 

Note: It is possible for the supervisor to trigger an interrupt artificially, by setting 
the appropriate flag in one of the registers ni_interrupt_cau.se/set or 
ni_interrupt_cause/set_green. Since this can be done for any interrupt, 
it is not documented under the triggering events for each interrupt. 

Also, since the ni_interrupt_clear and ni_interrupt_clear_green 
flags must be used to clear every interrupt once the required handling operations 
have been performed, this step is assumed, and is not listed under the handling 
guidelines for each interrupt. 
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B.1 Red Interrupts 

Red interrupts indicate a failure of the hardware, such as checksum violations 
and message format errors. They occur at unpredictable times relative to the in¬ 
struction stream and are usually irrecoverable. Det ermining the precise cause of 
a Red interrupt may require the use of the Diagnostic Network. 

The cause, clear, and set flags listed for each interrupt are found in the registers: 
n±__interrupt_cause 
ni_interrupt_clear 
ni__interrupt_set 


B.1.1 Internal Fault.Red Interrupt 

Flags: ni_cause/clear / se t_±nternal_£ aul t 

Cause: A fault has been detected in the NI chip. 

Effect: The effects are undefined and irrecoverable. 

Handling: No software-serviceable parts inside. Please report this fault to your 
applications engineer or systems manager for correction. 


B.1.2 CN Checksum Error, DR Checksum Error.Red Interrupt 

Flags: ni__cause/clear/set_cn_checksmn_error 

ni_cau.se/clear/set_dr__cliecksuni_error 

Cause: A message with a bad checksum value was received from either the 

Control Network or Data Network. This interrupt is signaled as 
soon as the bad checksum value is received by the NI. 

Effect: None. The received message(s) may still be read. However, they 

will almost certainly contain an error in either data or address. 

Handling: This interrupt indicates that a network chip (or the NI chip itself) 
has failed. The failed chip must be tracked down with the Diagnos¬ 
tic Network. Please report this fault to your applications engineer 
or system manager for correction. 


166 


NI Version 2.2 (CM-5E), June 1994 
Copyright © 1994 Thinking Machines Corporation 






Appendix B. NI Interrupts 


B.1.3 CN Hard Error 


Red Interrupt 


Flags: 

Cause: 

Effect: 


ni_cause/clear/set_cn_hard_error 
A hardware error occurred in the Control Network. 
The effects are undefined and irrecoverable. 


Handling: This interrupt indicates one of two things: either a hardware prob¬ 
lem in the Control Network, which must be located by use of the 
Diagnostic Network; or a serious software problem (specifically, a 
double trap forcing a processor (IU) reset). Please report this fault 
to your applications engineer or system manager for correction. 


B.1.4 MC Error, CMU Error .Red Interrupt 

Flags: ni_cause/clear/set_mc_error 

ni_cause/clear/set_cmu_error 

Cause: An interrupt is being signaled by either the memory controller, or 

by the CMU (cache and memory management unit). These two 
kinds of external interrupt are signaled to the microprocessor by 
way of the NI chip. 

Effect: None, aside from the interrupt itself. 

Handling: These interrupts continue to be signaled until they are cleared on the 
memory controller or CMU. 

Note: Unlike most NI interrupts, these two interrupts are not 
cleared by writing the corresponding ni_±nterrupt_clear flag. 
Instead, a flag on the memory controller or CMU must be reset. 

Nevertheless, it is legal to write a 1 to the ni_interrupt_clear 
flags for these interrupts. While this has no effect, it is permitted so 
that you can write uniform interrupt handler code. 
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B.1.5 BC Interrupt Red.Red Interrupt 

Flags: n±_cause/cleax/set_bc_interrupt_red 

Cause: The NI received a Red broadcast interrupt, and the broadcast inter¬ 

rupt enable flag ni_interxupt_xec_enable was set to 1. 

Effect: None, aside from the interrupt itself. 

Handlin g: This is a software-signaled interrupt. Your interrupt handler should 
detect and handle this interrupt as appropriate for your program. 


B.2 Orange Interrupts 

Orange interrupts indicate that the attention of the operating system is required, 
as in timer interrupts and broadcast interrupt messages. They occur at unpredict¬ 
able times relative to the instruction stream and do not destroy any information 
that might be needed to determine the cause of the interrupt. 

The cause, clear, and set flags listed for each interrupt are found in the registers: 
ni_interrupt_cause 
ni_±nterrupt_clear 
ni_±nterrupt_set 


B.2.1 Timer Interrupt.Orange Interrupt 

Flags: ni_cause/clear/aet_timer_interrupt 

Cause: The ni_time register is equal to the ni_±nterrupt_now register, 

and the timer interrupt flag ni_timei_ie flag is 1. 

Effect: None, aside from the interrupt itself. 

Handling: This interrupt is software-controlled, and should be handled by 
your interrupt handler. 
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B.2.2 Network Done Complete 


. Orange Interrupt 


Flags: 

Cause: 

Effect: 


ni_cause/clear/set_rdone_complete 

The ni_rdone_complete_ie flag is true, and a network-done 
operation has just completed (that is, the flag ni_router_ 
done_complete flag has been set). 

None, aside from the interrupt itself. 


Handling: This interrupt is software-controlled, and should be handled by 
your interrupt handler. It is intended to allow system code to do 
operations such as setting the ni_dr_message_count register to 
zero at the end of a network-done operation. 


B.2.3 BC Interrupt Orange 


Orange Interrupt 


Flags: 

Cause: 

Effect: 

Handling: 


n±__cause/clear/ set_bc_interrupt_orange 

The NI received a Orange broadcast interrupt, and the broadcast in¬ 
terrupt enable flag ni_interrupt_rec_enal>le was set to 1. 

None, aside from the interrupt itself. 

This is a software-signaled interrupt. Your interrupt handler should 
detect and handle this interrupt as appropriate for your program. 


B.3 Yellow Interrupts 

Yellow interrupts indicate a software error. They are usually irrecoverable, as 
they indicate that your program is doing something illegal. Sufficient informa¬ 
tion is retained in the NI to permit isolation of the cause of the interrupt, but it 
is not always possible to recover all information relating to the cause. Yellow 
interrupts are associated with particular instructions, but are not signaled at the 
exact point of the error, because of the loose Nl/microprocessor coupling. The 
cause, clear, and set flags listed for each interrupt are found in the registers: 
n±_±nterrupt_cause 
ni_interrupt_clear 
ni_infcerrupt_set 
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B.3.1 BC Interrupt Yellow.Yellow Interrupt 

Flags: ni_cause/clear/set_bc_interrupt_yellow 

Cause: The NI received a Yellow broadcast interrupt, and the broadcast in¬ 

terrupt enable flag nl_interrupt_rec_enable was set to 1. 

Effect: None, aside from the interrupt itself. 

Handling: This is a software-signaled interrupt. Your interrupt handler should 
detect and handle this interrupt as appropriate for your program. 


B.3.2 Bad Memory Access.Yellow Interrupt 

Flags: ni_cause/clear/set_bad_jmemory_access 

Cause: The NI would have signaled a Bus Error, but the flag 

ni_disable_bus_error was set to 1. 

Effect: Same as described for Bus Errors in Section B.5. 

Handling: Examine the ni_bad_addzess register to determine what 
memory transaction caused the error. 


B.3.3 COM Abstain Changed.Yellow Interrupt 

Flags: ni_cause/clear/set__com_abstain_changed 

Cause: The ni_com_abstain or ni_reduce_rec_absta±n flags were 

changed while the combiner send FIFO was not empty. 

Effect: The attempted change does not occur. Whether execution is allowed 

to continue depends on the interrupt handler. 

Handling: Your interrupt handler should decide whether to signal this as an 
error, or to recover from it quietly, perhaps displaying a warning 
message. 
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B.3.4 DR Count Negative .Yellow Interrupt 

Flags: ni_caus e/clear/ se t_dr_count_nega t i ve 

Cause: The combined value of all ni_dr_message_count registers in 

the Data Network has become negative, indicating a mismatch in 
the sending and/or receiving of Data Network messages. 

Effect: None, but this interrupt is signaled repeatedly until the situation is 

corrected. 

Handling: This may occur either when a failure in a Data Network or NI chip 
causes the annihilation of a message, or when an OS error causes 
a countable Data Network message to be sent out of its partition. 
This interrupt may also occur if two or more nodes in a paritition 
do not agree on which Data Network message tags are to be counted 
(that is, their ni_count_mask registers are not equal). 

To restore the Data Network to a proper state, make sure that the 
partition is empty of Data Network messages, and then set all the 
n±_dr_message_count registers in the partition to 0. 

Note: It may be that by the time the interrupt is signaled, the values 
of one or more of the ni_jdr_message__count registers will have 
changed. This may make it difficult to locate the error, since the 
sum of the ni_dx__me s sage_count registers may be positive by 
the time the interrupt is signaled. 


B.3.5 BC or COM Collision .Yellow Interrupt 

Flags: n±_cause/clear/set_bc_or_com_coll±s±on 

Cause: Three separate conditions cause this interrupt: 

■ Two NIs attempted to broadcast at the same time. 

* Two different combine operations signaled at the same time. 

■ Two NIs simultaneously attempted a broadcast interrupt. 

Effect: No combining or broadcast operations can proceed while the 

ni_cause_bc_or_com_collision flag is set. If the error was 
colliding broadcast interrupts, the broadcast is not signaled. 
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Han d ling : If the error was colliding combine messages, the messages are still 
in the combine send FIFO. The supervisor should take control of 
this FIFO and read out the messages to determine where the colli¬ 
sion occurred. If the error was colliding broadcast messages, the 
ni_bc_send_empty (or ni_sbc_send_empty) flags will be set 
to 0 in the contending processors. If the error was colliding broad¬ 
cast interrupts, the ni_interrupt_send_ok will be 0 in the 
processors that sent the colliding broadcast interrupts. 

The proper way to handle this interrupt is to set all the combine stop 
flags, then set the FIFO lock flags, then read out any remaining data 
values from the combine send FIFO. 

Note: When the ni_cleaz_bc_or_com_collis±on flag is writ¬ 
ten, all messages in the broadcast and supervisor broadcast send 
FIFOs disappear, and the ni_interrupt_send_ok flag is set to 1. 
None of the other FIFOs, either send or receive, are affected. 


B.3.6 Bad Relative Address .Yellow Interrupt 

Flags: n±_cause/clear/set_bad_relative_address 

Cause: An attempt was made to send a Data Network message with a rela¬ 

tive address that is illegal for the current partition. 

Effect: The message with the bad address is discarded and the appropriate 

xii_interface_Bend_ok flag is set to 0, indicating that the attempt 
to send the message failed. 


Handling: Your interrupt handler should decide whether to signal this as an 
error, or to recover from it quietly, perhaps displaying a warning 
message. 


B.3.7 Message Too Long .Yellow Interrupt 

Flags: ni_cause/clear/set_inessage_too_long 

Cause: An attempt was made to send a Data Network message with a 

length greater than is allowed for the interface in use. For each of 
the three send_f irst_long interfaces, this is the value of the 
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n±_longest_dr_message register. For the send_f irst register 
interfaces this is either the ni_longest_dr_message value or 
five words, whichever is less. 

Effect: The message with the bad address is discarded and the appropriate 

n±_interface_send_ok. flag is set to 0, indicating that the attempt 
to send the message failed. A bus error is also signaled. 

Handling: Your interrupt handler should decide whether to signal this as an 
error, or to recover from it quietly, perhaps displaying a warning 
message. 


B.4 Green Interrupts 

Green interrupts indicate the occurrence of common events for which the soft¬ 
ware has requested notification, such as the arrival of messages, the signaling of 
broadcast interrupts, arithmetic overflow in a scan, etc. There is one interrupt for 
each event, and each event’s interrupt can be enabled and disabled independently 
under the control of the supervisor. 

Depending on the type of event, the interrupt may or may not occur synchronous¬ 
ly with a particular instruction. No information is lost by a Green interrupt. 

The cause, clear, and set flags listed for each interrupt are found in the registers: 
n±_interrupt_cause 
ni_interrupt_clear 
ni_interrupt_set 


B.4.1 BC Interrupt Green 


Green Interrupt 


Flags: 

Cause: 

Effect: 

Handling: 


ni_cause/clear/set_bc_interrupt_green 

The NI received a Green broadcast interrupt, and the broadcast in¬ 
terrupt enable flag ni_interrupt_rec_enable was set to 1. 

None, aside from the interrupt itself. 

This is a software-signaled interrupt. Your interrupt handler should 
detect and handle this interrupt as appropriate for your program. 
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B.4.2 DR Receive Tag. Green interrupt 


Flags: ni_cause/clear/set_dr_rec_tag 

Cause: A message arrived at the front of a Data Network receive FIFO that 

has an interrupting tag (a tag corresponding to a set flag in the regis¬ 
ter ni__rec_±nterrupt_mask). 

Effect: None, aside from the interrupt itself. 

Handling: This interrupt is software-controlled, and should be handled by 
your interrupt handler. 


B.4.3 DR Receive AH Fail Down. Green interrupt 

Flags: ni_cause/cl ear /set_dr_r e c_al 1_£ al l_down 

Cause: An All Fall Down mode message arrived at the front of a Data Net¬ 

work receive FIFO, while nl_al l_f a 1 l_down_± e is 1. 

Effect: The first word read from the FIFO is the All Fall Down mode ad¬ 

dress word, which is used to determine the correct destination 
address for the message. The rec_length field contains the origi¬ 
nal length of the message (that is, not counting the address word), 
while the rec_length._lef t field contains the total length of the 
message counting the address word. 

Handling: Your handler should receive and store the message in such a way 
that it can later be resent to its correct destination. 


B.4.4 interface (DR, BC, COM, etc.) Receive OK ... Green Interrupt 

Flags: ni_cause/clear/set_bc_rec_ok 

ni__cause/cleax/set_sbc_rec_ok 
ni_cause/clear/set_com_rec_ok 
ni_cause/clear/set_dr_rec_ok 
ni_cause/clear/se t_ldr__rec_ok 
ni cause/clear/set rdr rec ok 
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Cause: A new message became available from the receive FIFO of one of 

the interfaces while the corresponding n±_interface_x ec_ok_ie 
flag was set to 1. 

Effect: While enabled, each of these interrupts is signaled once for each 

arriving message in the appropriate interface’s receive FIFO. 

Handling: This interrupt is software-controlled, and should be handled by 
your interrupt handler. (Typically, your handler reads the interrupt¬ 
ing message from the FIFO, but you can decide to do otherwise.) 


B.4.5 Global Rec (Sync, Global, or Supervisor) .... Green Interrupt 

Flags: ni_c aus e/clear/se t_sync_gl obal_r e c 

n±_cause/clear/set_global_rec 
ni_cause/clear/set_supervisor_global_rec 

Cause: One of the following events happened: 

A synchronous global operation completed with a result of 1, and 
the ni_aync_global_rec_ie flag is 1. 

The asynchronous global receive flag ni_global_rec changed 
from 0 to 1, and the nl_global_rec_ie flag is 1. 

The supervisor asynchronous receive flag ni_supervi- 
sor_g!obal_rec changed from 0 to 1, and the 
ni_supervisor_global_rec_ie flag is 1. 

Effect: None, aside from the interrupts themselves. 

Handling: These interrupts are software-controlled, and should be handled by 
your interrupt handler. 


B.4.6 Com Receive Empty. Green Interrupt 

Flags: n±_cause/clear/set_com_rec_empty 

Cause: The combine receive FIFO became empty while the empty receive 

FIFO interrupt flag ni_com_rec_empty_±e is 1. 
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Effect: None, aside from the interrupt itself. 

Handling: This interrupt is software-controlled, and should be handled by 
your interrupt handler. 


B.4.7 Scan Overflow . Green Interrupt 

Flags: ni_cause/clear/set_scan_overflow 

Cause: The first word of a scan or reduce message that suffered arithmetic 

overflow was read from the combine receive FIFO, and the 
ni_scan_over £ low_±e interrupt enable flag is 1. This can only 
happen if the message combiner is a signed or unsigned addition. 

Effect: None. The arrived message may be read normally. 

Han dlin g: Your interrupt handler should decide whether to signal this as an 
error, or to recover from it quietly, perhaps displaying a warning 
message. 


B.4.8 DP Error (Vector Unit Error) . Green Interrupt 

Flags: ni__cau.se/clear/set_dperr 

Cause: An interrupt has been signaled by the node’s memory controller 

(the vector units in CMs so equipped). These interrupts are sent to 
the PE by way of the NI. 

Effect: This interrupt will continue to be signaled until it is cleared both on 

the memory controller and in the NI. 

Handling: This interrupt is introduced in Version 2.2 so that the vector units, 
integrated into the memory controller chips, can signal green inter¬ 
rupts. Both the NI ni_interrupt_clear_green flag and the 
corresponding flag on the memory controller (or VU) must be 
written to clear this interrupt. 
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B.4.9 Send FIFO Empty (Data Network Only). Green Interrupt 

Flags: ni_cause/clear/set__sf if o_empty 

Cause: The ni_sf if o_empty_ie flag is set, and a send FIFO in one of 

the Data Network interfaces (DR, LDR, RDR) has become empty. 

Effect: None. The arrived message may be read normally. 

Handling: This interrupt is intended as an aid in sending Data Network mes¬ 
sages; in particular, the supervisor can wait until this condition 
occurs before sending messages, rather than attempting several 
failed sends when the Data Network is congested. 


B.4.10 LDR/RDR Tag, LDR/RDR User Tag . Green Interrupt 

Flags: ni_cause/clear/set__ldr_tag 

ni_cause/clear/set_rdr_tag 
ni_cause/clear/set__ldr_user_tag 
ni_cause/clear/set__rdx_user_tag 

Cause: A message arrives at the front of the left (or right) Data Network 

receive FIFO, having a tag that corresponds to a 1 bit in the register 
n.i_rec_interrupt_mask (for ldr/rdr_tag interrupts) or in 
nl_user_rec_interrupt_mask (for the ldr/rdr_user_tag 
interrupts). 

For the user_tag interrupts, not only must the appropriate tag 
mask be set in the ni_user_r ec_inter ruptjnask, but the same 
bit must be cleared in the ni_rec_interrupt_mask register. 

Effect: None. The arrived message may be read normally. 

Handling: These interrupts are intended as an aid in receiving Data Network 
messages. Your interrupt handler should determine the appropriate 
action to take to receive the tagged message that signaled the inter-. 
rapt. 
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B.5 Bus Errors 

Bus Errors indicate that a bus transaction cannot be completed, as in an attempt 
to read a virtual address that does not correspond to a register, or to write a mes¬ 
sage that doesn’t conform to protocol. Bus Errors are signaled asynchronously 
and are usually irrecoverable. Bus Errors are distinct from segmentation viola¬ 
tion errors, which result from attempting to read an unmapped virtual address, 
and are signaled synchronously with the offending instruction. 

The cause and clear flags listed for each interrupt are found in these registers: 
ni_interrupt_cause ni_interrupt_clear 

B.5.1 Bad Memory Access .Bus Error 

Flags: ni_cause/clear / se t_bad_xnemory_acces s 

Cause: Bus Errors are signaled for number of reasons, including: 

■ attempting to read a read-protected address 

■ attempting to write a write-protected address 

* attempting to read or write a value that does not fit in a register 

■ attempting to read or write an address that is not a register 

Note: If the flag ni_disable_bus_error is set, Bus Errors are signaled as 

a Yellow Interrupt (see Section B.3.2 above). 

Some specific examples of Bus Error causes are: 

Bus Errors caused by reads or writes: 

■ reading or writing a supervisor-only register from the user area 

* reading the n±_interface_x e c register of an empty receive FIFO 

■ attempting to read a doubleword from a FIFO that has only a word left, or 
attempting to use a doubleword operation to write a singleword message 

■ writing the send_f irst register of a network interface while there is an 
incomplete message pending in the send FIFO 

■ writing the send register of a network interface without having first writ¬ 
ten a value to the corresponding send_£±rst register 
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■ writing a message to any of the Data Network’s send__f irst registers 
with a length value that is greater than either five words or the value of the 
register ni_longest_dr_message, whichever is less. 

■ writing a message to any of the Data Network’s send_f irst_long reg¬ 
isters with a length value that is greater than the value of the register 
ni_longe s t_dr_me s s age. 

Bus Errors caused by sending a message: 

■ attempting to send a message longer than the entire send FIFO 

■ attempting to send a message via a network interface for which the corre¬ 
sponding abstain flag is set 

■ attempting to send a user message with a supervisor-reserved tag 

■ attempting to send or receive a message through an excluded Data Net¬ 
work interface 

■ attempting to send a combine message with an illegal combiner or pattern 
value 

■ attempting to send a network-done message with a length greater than 1, 
or attempting to send any network-done message while the ni_.net - 
work_done flag is 0 or the ni_com_abstain flag is 1 

■ attempting to send a synchronous global message or to change the 
n±_sync_global_abstain flag while the ni_sync_gl obal_com - 
plete flag is 0 

Bus Errors caused by other operations: 

■ attempting to start a flush operation while the ni_f lush_complete flag 
is 0 

■ attempting to start a configuration operation while the ni__conf igura- 
tion_complete flag is 0 

■ attempting to send a broadcast interrupt while the ni_inter- 
rupt_send_ok flag is 0 

■ attempting to write a value to the n±_interface_x&c register when the re¬ 
ceive FIFO is full. 


NI Version Z2 (CM-5E), June 1994 

Copyright © 1994 Thinking Machines Corporation 


179 


NI Programmer’s Handbook 


Effect: The address, size, and type of the offending memory transaction is 

stored in the ni_bad_address register. 

Any data written by the offending transaction is lost. Any side 
effects that would have been triggered by the offending transaction 
(such as the initiation of a synchronous global operation) do not oc¬ 
cur. In particular, an attempted doubleword read from a receiving 
FIFO containing only one word will not result in popping the word. 

Handling: Examine the ni_bad_address register to determine what 
memory transaction caused the error. 
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This appendix describes the important C macros and constants defined by the 
CMNA software layer (that is, those relating to the NI chip itself). 


C.1 Generic Variables and Macros 

To determine the address of a node, and its place within its partition, use these 
variables: 


int CMNA_self_address - Relative address of current node, 

int CMNA_partition_size - Number of nodes in partition. 

These are the macros used to examine fields of the ni._interface_ata.txia regis¬ 
ter (but not die status_all register) for any interface that has such a register: 

Field Name: Macros Used to Read Value of Field: 


ni_send_ok 
ni_send_space 
ni_s end_emp ty 
ni_r ec_ok 
ni_r ec_length 
ni_r ec_length_lef t 


SEND_OK ( status_value) 

SEND_S PACE ( status_yalne) 
s end_empt Y ( status_value) 
receive_OK [status_value) 

RECEIVE_LENGTH [status_value) 
receive_length_left ( status_yalue) 


For interfaces that have an abstain flag, there is a pair of macros that can be used 
to read and write the value of the flag: 


value = CMNA_read_abstain_f lag (register_address) ; 
CMNA_wr ite_abstain_f lag ( register_address, value) ; 
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For both macros, register_address is a symbolic constant giving the address of 
the abstain flag register (this is defined separately for each interface that has such 
a register). 

For the write macro, value is the new value (0 or 1) to be written to the flag. 


C.2 Data Network Constants and Macros 


Send and Receive Register Macros 

The send_f irst registers for the Data Network interfaces are accessed via the 
macros below: 

Register Name: _ Macros Used to Write First Vaiue of Message: _ 

ni_dr_send_first CMNA_dr_send_f irst {_long} {tag, length, value) 

CMNA_dx_send_f ixst_double {_long} (tag, length, value) 
ni_ldr_send_first CMNA_ldr_send_first (_long) (tag, length, value) 

CMNA_ldr_send_first_double{_long} (tag, length, value) 
ni_rdr_send_first CMNA_rdr_send_first {_long} (tag, length, value) 

CMNA_rdr_send_first_double{_long} (tag, length, value) 

The length argument in each case is the total length in words of the message to 
be sent (excluding the address word), and the tag argument is the message’s tag 
value. 


The send and rec registers of the Data Network interfaces can be written to and 
read from by the generic register macros in Section C.l, and by the following 
special-purpose macros: 


Register Name: Macros Used to Access Register: 


ni dr send 


ni ldr send 


ni_ldr_recv 


CMNA_dr_s end_wor d (word_value) 

CMNA_dr_send_f loat (float_value) 
CMNA_dr_send_double (double,_value) 
CMNA_ldr_send_word (word_value) 
CMNA_ldr_send__f loat (float_value) 
CMNA_ldr_send_double (double_value) 
word_value = CMNA_ldr_receive_word(); 
float_value = CMNA_ldr_receive_float(); 
double value « CMNA ldr receive double(); 
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Register Name: _ Macros Used to Access Register: _ 

ni__r dr_s end CMNA_r dx_s end_wor d ( word_value ) 

CMNA_rdr_send_f loat (float_yalue ) 
CMNA_xdx_send_double {double_value) 
ni_rdr_rec word_yalue = CMNA_rdr_receive_word() ; 

float_value = CMNA_rdr_receive_float () ; 
double value - CMNA rdr receive doublet) 


Status Register Macros 

The values of the Data Network status registers can be obtained by using these 
macros: 

int dr_status = CMNA_dr_send_status(); 
int ldx_status = CMNA_ldr_status(); 
int rdr_status = CMNA_rdr_status 0 ; 

You can extract the fields of the status registers by applying these macros: 

Register/Field Name: _ Macros Used to Access Fields: _ 

ni_dx_status 
ni_send_ok 
ni_send_space 
ni_send_state 
ni_xec_state 
ni_r out er _done_c ompl e t e 
ni_ldr_status 
ni_send_ok 
ni_s end._space 
ni_rec_ok 
ni_ldr_r ec_tag 
ni_r ec_length 
ni_rec_length_lef t 
ni_rdr_status 
ni_send_ok 
ni_send_space 
ni_rec_ok 
ni_r dr_r ec_tag 
ni_r ec_length 
ni_r ec_length_left 
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SEND_OK (dr_status) 

SEND_S PACE(dx_status) 

DR_S END_S TATE(dr_status) 
DR_RECEIVE_STATE(dr_status) 
DR_ROUTER_DONE(dr_status) 

SEND_OK(ldx_status) 

S END_S PACE(ldr_status) 
RECEIVE_OK(ldr_status) 

RECEIVE_TAG(ldr_status) 

RECEIVE_LENGTH(ldr_s tatus) 

RECEIVE_LENGTH_LEFT(ldr_status) 

SEND_OK(rdx_status) 

S END_S PACE(xdx_s tatus) 

RECEIVE_OK(r dx_s tatus) 

RECEIVE_TAG(xdr_status) 

RECEIVE_LENGTH(r dr_s tatus) 
RECEIVE LENGTH LEFT(rdr status) 
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Message Length Limit 

The maximum length of a Data Network message (not counting the address word 
attached in sending it) is given by the constant 

MAX ROUTER MSG WORDS 


C.3 Broadcast Interface Constants and Macros 

Send and Receive Register Macros 

The send_f irst register for the broadcast interface is accessed via the macros 
listed here: 

Register Name: _ Macros Used to Write First Value of Message: _ 

ni_bc_send_first CMNA_bc_send_first {length, value) 

CMNA_bc_s end_f i r s t_doub 1 e ( length, value) 

The send and rec registers of the broadcast interface can be written to and read 
from by the following special-purpose macros: 

Register Name: Macros Used to Access Register: 

ni_bc_send . CMNA_bc_send_word (word_value) 

CMNA_bc_send_f loat [floatjyalue) 
CMNA_bc_send_double ( double_value) 

ni_bc_recv word__value = CMNA_bc_receive_word () ; 

float_value = CMNA__bc_receive_f loat () ; 
double_value = CMNA_bc_receive double (); 


Status Register Macros 

The value of the broadcast interface status register can be obtained by using this 
macro: 

int bc_status = CMNA be status(); 
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You can extract the fields of the status register by applying the following macros: 

Register/Field Name: Macros Used to Access Fields: _ 


ni_bc_s tatus 

ni_send_ok SEND_OK(bc_status) 

ni_send_space SEND_SPACE(bc_status) 

ni_send_empty SEND_EMPTY(bc_status) 

ni_rec_ok RECEIVE_OK(bc_status) 

ni_r ec_length._left BC_RECEIVE_LENGTH(bc_status) 


Abstain Register Macros 

The broadcast abstain register contains a single flag bit, which can be read and 
written using the generic abstain bit operations described in Section C.l. 

Register/Field Name: Macros Used to Access Fields: 

ni_bc_con.tr ol 

ni_rec_abstain value=CMNA_read_abstain_flag 

(bc_control_reg); 
CMNA_writ e_abs t ain_flag 

(bc_control_reg,value); 


Message Length Limit 

The maximum length of a broadcast message is given by the constant 

MAX BROADCAST MSG WORDS 
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C.4 Combine Interface Constants and Macros 

Send and Receive Register Macros 

The send_first register for the combine interface is accessed via the macros 
below: 

Register Name: _ Macros Used to Write First Value of Message: _ 

ni_c om_s end_first CMNA_com_send_first 

( combiner, pattern, length, value) 
CMNA_c oin_s end_f ir st_double 

( combiner, pattern, length, value) 

For scan operations, the combiner argument can be any one of the constants 

ADD_SCAN MAX_SCAN OR_SCAN 

UADD_SCAN XOR_SCAN 

and the pattern argument can be any one of the constants 

SCANJBACKWARD SCAN_FORWARD SCAN_REDUCE 

For network-done operations there is a unique combiner and pattern pair: 

combiner: as s ERT_ROUTER_DONE 
pattern: scan_router_done 


The send and xec registers of the combine interface can be written to and read 
from by the generic register macros in Section C.l, and by the following special- 
purpose macros: 

Register Name: _ Macros Used to Access Register: _ 

ni_com_send CMNA_com_send_word ( word_value) 

CMNA_com_send_f loat (float_value) 
CMNA_com_send_do-uble {double_value) 
ni_com_recv word_value - CMNA_com_receive_word(); 

float_value = CMNA_conn_receive_f loat () ; 
double_value 

= CMNA_com receive double(); 
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Message Length Limit 

The maximum length of a combine message (with the exception of network-done 
messages, which are always 1 word) is given by the constant 

MAX COMBINE MSG WORDS 


Segment Start Register Macros 

The ni_scan_start register is accessed by the following special purpose mac¬ 
ros: 

Register Name: _ Macros Used to Access Register: _ 

ni_scan_start CMNA_set_segment_start {value) 

value = CMNA_segment_star t () ; 


Status Register Macros 

The value of the combine interface status register can be obtained by using the 
macro 


int com_status = CMNA_com_status(); 

You can extract the fields of the status register by applying the following macros: 

Register/Field Name: Macros Used to Access Fields: 


ni_com_status 
ni_send_ok 
ni_send_space 
ni_send_empty 
ni_r ec_ok 
ni_rec_length 
ni_rec_length_left 
ni com scan overflow 


SEND_OK(com_status) 

SEND_SPACE(com_status) 
SEND_EMPTY(com_status) 
RECEIVE_OK(com_status) 
RECEIVE_LENGTH(Com_s tatus) 
RECEIVE_LENGTH_LEFT(com_status) 
COMBINE OVERFLOW(com status) 
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Abstain Register Macros 

The combine abstain register contains two single-bit flags, which can be read and 
written by the macros listed below: 

Register/Field Name: Macros Used to Access Fields: 


ni_com_control 

ni_rec_abstain value=CMNA_read_abstain_flag 

(com_control_reg) ; 
CMNA_wxite_abstain_flag 

(com__contiol_x eg, value) ; 


ni_reduce_rec_abs tain 

value=CMNA_read_xec_abstain_flag(coin_contxol_reg ); 
CMNA_wxite_xec_abstain_f lag (corn_contzol_x eg, value) ; 


C.5 Global Interface Constants and Macros 

Synchronous Global Register Macros 

The synchronous global registers are read and written by the following macros: 
Register Name: _ Macros Used to Access Register: _ 

ni_sync_global_send CMNA_or_global__sync_bit (value) 

ni_sync_global 

ni_sync_global_complete value = 

CMNA_global_sync_complete () 
ni_sync_global_rec value = 

CMNA_global_sync_rec() 

ni_sync_global_abstain 

value® CMNA_r ead_abstain_flag 

(sync_global_abstain_reg); 
CMNA_wr i t e_ab stain__f lag 

(sync_global_abstain_r eg, value) 
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Asynchronous Global Register Macros 

The two flags of the asynchronous global register are read and written by these 
macros: 

Reglster/F>ag Name: _ Macros Used to Access Register: _ 

ni_async_global 

ni_global_send CMNA_or_global_async_bit (value) 

ni_global_rec value = CMNA_global_async_read() 
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For ease of reference, here are the low-level programming constants defined in 
the header files cmsys/ni_constants.h, and cmsys/n±_defines.h (see 
Appendix H), grouped by register and field. 

Note for C Programmers: These constants are defined as raw, unsigned integer 
values. If you use them in C code, you must recast them as pointer values of type 
(unsigned *). Otherwise, the C compiler will treat them as integers, possibly 
causing “illegal pointer operation” errors. 


=== Send First Register Constants »== 
Field Offsets: 

SF_FIFO_OFFSET (12) 

AUXILIARY_START_P (3) 

Length Constant: NI_SEND_FIRST_L (32) 

Interface Number constants: 


DATA_ROUTER_FIFO (1) 
LEFT_DR_FIFO (6) 
RIGHT_DR_FIFO (7) 
USER_BC_FIFO (3) 
SUPERVISOR_BC_FIFO (4) 
COMBINE FIFO (5) 


=»==» Auxiliary Data Field Constants ==•« 

- DR/LDR/RDR Interface - 

NI_DR_SEND_AUXILIARY_ADDRESS_MODE_P (8) 

RELATIVE (0) 

PHYSICAL (1) 

NI_DR_SEND_AUXILIARY_TAG_P (4) NI_DR_TAG_L (4) 

NI_DR_SEND_AUXILIARY LENGTH P (0) NI DR LENGTH L (4) 
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=== Auxiliary Data Field Constants, 

- BC/SBC Interface - 

NI BC SEND AUXILIARY LENGTH P (0) 


cont. == = 

(no length constant) 


- COM Interface - 

NI_COM_SEND_AUXILIARY_PATTERN_P (7) 
NI_COM_SEND_PATTERN_L (2) 
SCAN_ROUTER_DONE (0) 

S CAN_BACKWARD (1) 

SCAN_FORWARD (2) 

SCAN_REDUCE (3) 

NI_COM_SEND_AUXILIARY_COMBINER_P (4) 
NI_COM_SEND_COMBINER_L (3) 

OR_SCAN (0) 

ADD_SCAN (1) 

XOR_SCAN (2) 

UADD_SCAN (3) 

MAX_SCAN (4) 

AS S ERT_ROUTER_DONE (5) 

NI_COM__S END_AUXIL I ARY_LENGTH_P {0) 

NI COM SEND LENGTH L (4) 


=== Interface send/receive FIFO size limits === 
MAX_ROUTER_MSG_WORDS (5) 

MAX_COMBINE_MSG_WORDS (5) 

MAX_BROAD CAS T_MS G_WORDS (4) 

MAX SBC MSG WORDS (4) 


===== Send Registers 

NI_DR_SEND_A 

NI_LDR_SEND_A 

NI_RDR_SEND_A 

NI_BC_SEND_A 

NI_SBC_SEND_A 

NI_COM_SEND_A 

NI SEND L (32) 


(NI_BASE | 0x0230) 
(NI_BASE 1 0x0c30) 
(NI_BASE | 0x0e30) 
(NI_BASE | 0x0630) 
(NI_BASE | 0x0830) 
(NIJBASE 1 0x0a30) 


=== Receive Registers === 


NI_DR_RECV_A 

NI_LDR_RECV_A 

NI_RDR_RECV_A 

NI_BC_RECV_A 

NI_SBC_RECV_A 

NI_COM_RECV_A 

NI REC L (32) 


(NI_BASE 
(NI_BASE 
(NI_BASE 
(NI_BASE 
(NI__BASE 
(NI BASE 


0x0220) 
0x0c20) 
0x0e20) 
0x0620) 
0x0820) 
0x0a20) 
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(NI_BASE | 0x0200) 

(NI_BASE + 0x0250) /* 2.2 */ 
(NI_BASE + 0x0260) /* 2.2 */ 
(NI_BASE | OxOcOO) 

(NI_BASE + 0x0c50) /* 2.2 */ 
(NI_BASE + 0x0c60) /* 2.2 */ 
(NI_BASE | OxOeOO) 

(NI_BASE + 0x0e50) /* 2.2 */ 
(NIJ3ASE + 0x0e60) /* 2.2 */ 
NI_STATUS_LONG_L (28) /* 2.2 */ 

NI_BC_STATUS_A (NI_BASE | 0x0600) 

NI_SBC_STATUS_A (NI_BASE | 0x0800) 

NI_BC_STATUS_L (11) 

NI_COM_STATUS_A (NI_BASE | OxOaOO) 

NI_COM_STATUS_L (21) 

NI_STATUS_L (25) 

Field Constants: 

NI_SEND_SPACE_P (0) NI_S END_S PACE_L (4) 

NI_REC_OK_P (4) NI_REC_OK_L (1) 

NI_SEND_OK_P (5) NI_SEND_OK_L (1) 

NI_ROUTER_DONE_COMPLETE_P(6) NI_ROUTER_DONE_COMPLETE_L (1) 
NI_SEND_EMPTY_P (6) NI_SEND_EMPTY_L (1) 

NI_REC_LENGTH_LEFT_P (7) NI_REC_LENGTH_LEFT_L (4) 

'NI_BC_REC_LENGTH_LEFT_LONG_L (7) /* 2.2 */ 
NI_REC_LENGTH_P (11) NI_REC_LENGTH_L (4) 

NI_DR_REC_TAG_P (15) NI_DR_REC_TAG_L (4) 

NI_COM_SCAN_OVERFLOW_P (20) NI_COM_SCAN_OVERFLOW_L (1) 
NI_DR_S END_S TATE_P (21) NI_DR_SEND_STATE_L (2) 

NI_DR_REC_STATE_P (23) NI_DR_REC_STATE_L (2) 

/* 2.2 */ 

NI_SEND_SPACE_LONG_P (0) NI_S END_S P ACE_LONG_L (5) 

NI_REC_OK_LONG_P (5) NI_REC_OK_LONG_L (1) 

NI_SEND_OK_LONG_P (6) NI_SEND_OK_LONG_L (1) 

NI_ROUTER_DONE_COMPLETE_LONG_P (7) 
NI_ROUTER_DONE_COMPLETE_LONG_L (1) 

NI_REC_LENGTH_LEFT_LONG_P (8) NI_REC_LENGTH_LEFT_LONG_L (5) 
NI_REC_LENGTH_LONG_P (13) NI_REC_LENGTH_LONG_L (5) 

NI_DR_REC_TAG_LONG_P (18) NI_DR_REC_TAG_LONG_L (4) 

NI_DR_SEND_STATE_LONG_P (24) NI_DR_SEND_STATE_LONG_L (2) 

NI DR REC STATE LONG P (26) NI DR REC STATE LONG L (2) 


=== Status Register ===== 

NI_DR_STATUS_A 

Ni:_DR_STATUS_ALL_A 

NI_DR_STATUS_LONG_A 

NI_LDR_STATUS_A 

NI_LDR_STATUS_ALL_A 

NI_LDR_STATUS_LONG_A 

NI_RDR_STATUS_A 

NI_RDR_STATUS_ALL_A 

NI_RDR_STATUS_LONG_A 

NI XDR STATUS L (19) 
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=== Control Registers === 

Nl_BC_CONTROL_A (NI_BASE | 0x0610) 

NI_SBC_CONTROL_A (NI_BASE | 0x0810) 

NI_BC_CONTROL_L (1) 

NI_COM_CONTROL_A (NI_BASE | OxOalO) 

NI_COM_CONTROL_L (2) 

NI_CONTROL_L (2) 

Field Constants: 

NI_REC_ABSTAIN_P (0) NI_REC_ABSTAIN_L (1) 

NI_REDUCE_REC_ABSTAIN_P (1) NI_REDUCE_REC_ABSTAIN_L (1) 

=== Private Registers === 

NI_DR_PRIVATE_A (NI_BASE | 0x0208) 

NI_DR_PRIVATE_L (10) 

NI_LDR_PRIVATE_A (NI_BASE | 0x0c08) 

NI_RDR_PRIVATE_A (NI_BASE | 0x0e08) 

NI__XDR_PRIVATE_L (6) 

NI_BC_PRIVATE_A (NI_BASE | 0x0608) 

NI_SBC_PRIVATE_A (NI_BASE | 0x0808) 

NI_BC_PRIVATE_L (5) 

NI_COM_PRIVATE_A (NI_BASE | 0x0aO8) 

NI_COM_PRIVATE_L (18) 

NI_PRIVATE_L (18) 

Field Constants: 

NI_REC_OK_IE_P (0) NI_REC_OK_IE_L (1) 

NI_LOCK_P (1) NI_LOCK_L (1) 

NI_REC_STOP_P (2) NI_REC_STOP_L (1) 

NI_REC_FULL_P (3) NI_REC_FULL_L (1) 

NI_SEND_ENABLE_P (4) NI_SEND_ENABLE_L (1) 

NI_BC_SEND_ENABLE_P (4) NI_BC_SEND_ENABLE_L (1) 

NI_COM_SCAN_OVERFLOW_IE_P(4) NI_COM_SCAN_OVERFLOW_IE_L (1) 
NI_DR_REC_ALL_FALL_DOWN_P(5) NI_DR_REC_ALL_FALL_DOWN_L (1) 
NI_COM_REC_EMPTY_IE_P (5) NI_COM_REC_EMPTY_IE_L (1) 

NI_ALL_FALLJDOWN_IE_P (6) NI_ALL_FALL_DOWN_IE_L (1) 

NI_ALL_FALL_DOWN__ENABLE_P(7) NI_ALL_FALL_DOWN_ENABLE_L (1) 
NI_COM_SEND_LENGTH_P (8) - NI_COM_SEND_LENGTH_L (4) 

NI_COM_SEND_COMBINER_P (12) NI_COM_SEND_COMBINER_L (3) 
NI_COM_SEND_PATTERN_P (15) NI_COM_SEND_PATTERN_L (2) 

NI_COM_SEND_START_P (17) NI COM SEND START L (1) 
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=== Global and System Registers === 

NI_INTERRUPT_CAUSE_A (NI_BASE | 0x0000) 

NI_INTERRUPT_SET_A (NI_BASE + 0x0190) /* 2.2 */ 

NI__CAUSE__INTEENAL_FAULT_P (0) 

NI_CAUSE_MC_ERROR_P (1) 

NI_CAUSE_CMU_ERROR_P (2) 

NI_CAUSE_BC_INTERRUPT_RED_P (3) 

NI_CAUSE_CN_CHECKSUM_ERROR_P (4) 

NI__CAUS E_CN_HARD_ERROR_P (5) 

NI_CAUSE_DR_CHECKSUM_ERROR_P (6) 

NI_CAUSE_TIMER__INTERRUPT_P (7) 

NI_CAUSE_BC_INTERRUPT_ORANGE_P (8) 
NI__CAUSE_BC_INTERRUPT_YELLOW_P (9) 
NI__CAUSE_BC_OR_COM_COLLISION_P {10) 

NI_CAUS E_COM_ABSTAIN_CHANGED_P (11) 
NI__CAUSE_DR_COUNT_NEGATIVE_P (12) 
NI_CAUSE__BAD_RELATIVE_ADDRESS_P (13) 
NI_CAUSE_BAD_MEMORY_ACCESS_P (14) 

NI_CAUSE__MESSAGE_TOO_LONG_P (15) /* 2.2 */ 
NI_CAUSE_RDONE_COMPLETE_P (16) /* 2.2 */ 

NI_INTERRUPT_L (1) 

NI_INTERRUPT_CAUSE_GREEN_A (NI_BASE | 0x0008) 

NI_INTERRUPT_SET_GREEN_A (NI_BASE + 0x0198) /* 2.2 */ 

NI__CAUSE_BC_INTERRUPT_GREEN_P (0) 

NI_CAUSE_SCAN_OVERFLOW_P (1) 

NI__CAUSE_BC_REC_OK_P (2) 

NI_CAUSE_SBC_REC_OK_P (3) 

NI_CAUSE_COM_REC_OK_P (4) 

NI_CAUSE_COM_REC_EMPTY_P (5) 

NI_CAUSE_SYNC_GLOBAL_REC_P (6) 

NI_CAUSE_GLOBAL_REC_P (7) 

NI_CAUSE_SUPERVISOR_GLOBAL_REC_P (8) 

NI_CAUSE_DR_REC_OK_P (9) 

NI_CAUSE_LDR_REC_OK_P (10) 

NI_CAUSE_RDR_REC_OK_P (11) 

NI_CAUSE_DR_REC_TAG_P (12) 

NI_CAUSE_DR_REC_ALL_FALL_DOWN_P (13) 

NI_CAUSE_LDR_REC_TAG_P (14) /* 2.2 */ 

NI_CAUSE_RDR_REC_TAG_P (15) /* 2.2 */ 
NI_CAUSE_LDR_USER_REC_TAG_P (16) /* 2.2 */ 
NI_CAUSE_RDR_USER_REC_TAG_P (17) /* 2.2 */ 

NI_CAUSE_SFIFO_EMPTY (18) /* 2.2 */ 

NI_CAUSE_DPERR (19) /* 2.2 */ 

NI INTERRUPT L (1) 
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NI__INTERRUPT_LEVEL_A 
NI_INTERRUPT_LEVEL_L (32) 

NI INTERRUPT LEVEL COLOR L (8) 


NI_LONGEST_DR_MES SAGE_A 

NI_US ER_RE C_INTERRUPT_MAS K_A 

NI_PHYSI CAL_SELF_A 

NI_PARTITION_BASE_A 
NI_PARTITION_SIZE_A 
NI_PHY SI CAL__ADDRE S S_L (20) 

NI_CHUNK_TABLE_ADDRES S_A 
NI_CHUNK_TABLE_ADDRES S_L (6) 

NI_CHUNK_TABLE_DATA_A 
NI_CHUNK_TABLE_DATA_L (8) 

NI_CHUNK_SIZE_A 
NI_CHUNK_SIZE_L (3) 

NI_DR_MES SAGE_COUNT_A 
NI_DR_MES SAGE_COUNT_L (32) 

NI_COUNT_MASK_A 

NI_REC_INTERRUPT_MASK_A 

NI_USER_TAG_MASK_A 
NI_TAG_MASK_L (16) 

NI_TIME_A 
NI_TIME_L (32) 

NI_CONFIGURATION_A 
NI_CONEIGURATION_L (5) 

NI_INTERRUPT_SEND_A 
NI_INTERRUPT_SEND_L (5) 

NI_SERIAL_NUMBER_A 
NI_SERIAL_NUMBER_L (32) 

NI_SYNC_GLOBAL_A 

NI_S YN C_GLOB AL_RE C_P (0 ) 

NI_SYNC_GLOBAL_REC__L (1) 

NI SYNC GLOBAL COMPLETE P(l) 


(NI_BASE [ 0x0010) 


(NI_BASE + 0x0160) /* 2.2 */ 
(NI_BASE + 0x0168) /* 2.2 */ 


(NI BASE 1 

0x0018) 


(NI_BASE | 

0x0020) 

(NI_BASE | 

0x0028) 

(NI BASE 1 

0x0030) 


(NI_BASE | 

0x0038) 

(NI BASE 1 

0x0040) 


(NI BASE 1 

0x0048) 


(NI BASE 1 

0x0050) 


(NI_BASE | 

0x0058) 

(NI_BASE | 

0x0060) 

(NI_BASE | 

0x0070) 

(NI BASE 1 

0x0078) 


(NI_BASE | 

0x0080) 

(NI BASE 1 

0x0088) 


(NI_BASE | 

0x0090) 
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NI_SYNC_GLOBAL_COMPLETE_L (1) 

NI_SYNC_GLOBAL_L (2 ) 

NI_SYNC_GLOBAL_ABSTAIN_A (NI_BASE | 0x0098) 

NI_SYNC_GLOBAL_ABSTAIN_L (1) 

NI_COM_FLUSH_SEND_A (NIJBASE | 0x0OaO) 

NI_FLUSH_SEND_L (1) 

NI_ASYNC_GLOBAL_A 
NI_GLOBAL_SEND_P (0) 

NI_GLOBAL_REC_P (1) 

NI_GLOBAL_L (2) 

NI_ASYNC_SUP_GLOBAL_A (NI_BASE | 0x0ObO) 

NI_SUPERVISOR_GLOBAL_SEND_P (0) 
NI_SUPERVISOR_GLOBAL_SEND_L (1) 
NI_SUPERVISOR_GLOBAL_REC_P {1) 

NI_SUPERVISOR_GLOBAL_REC_L (1) 

NI_GLOBAL_L (2) 

NI_HODGEPODGE_A (NI_BASE | 0x00b8) 

NI_GLOBAL_REC_IE_P (0) 

NI_GLOBAL_REC_IE_L (1) 

NI_SUPERVISOR_GLOBAL_REC_IE_P (1) 

NI_SUPERVISOR_GLOBAL_REC_IE_L (1) 

NI_FLUSH_COMPLETE_P (2) 

NI_FLUS H_COMPLETE_L (1) 

NI_INTERRUPT_SEND_OK_P (3) 

NI_INTERRUPT_SEND_OK_L (1) 
NI_CONFIGURATION_COMPLETE_P (4) 

NI_CONFIGURATION_COMPLETE_L (1) 
NI_INTERRUPT_REC_ENABLE_P (5) 
NI_INTERRUPT_REC_ENABLE_L (1) 
NI_SYNC_GLOBAL_REC_IE_P (6) 

NI_SYNC_GLOBAL REC_IE_L (1) 

NI_TIMER_IE_P {7) 

NI_TIMER_IE_L (1) 

N 1 _CN_ST° P_S END_P (8) 

N I _ c N_STOP_SEND_ L (1) 

NI_DISABLE_BUS_ERROR_P (9) /* 2.2 */ 
NI_DISABLE_BUS_ERROR_L (1) /* 2.2 */ 
NI_LDR_REC_TAG_IE_P (10) /* 2.2 */ 
NI_LDR_REC_TAG_IE_L (1) /* 2.2 */ 
NI_RDR_REC_TAG_IE_P (11) /* 2.2 */ 

NI RDR REC TAG IE L (1) /* 2.2 */ 


(NI_BASE | 0x00a8) 
NI_GLOBAL_S END_L (1) 

NI GLOBAL REC L (1) 
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NI_LDR_USER_REC_TAG_IE_P (12) /* 2.2 */ 
NI_LDR_USER_REC_TAG_IE_L (1) /* 2.2 */ 
NI_RDR_USER_REC_TAG__IE_P (13) /* 2.2 */ 
NI_RDR_USER_REC_TAG_IE_L (1) /* 2.2 */ 

NI_MSG_TOO_LONG_IE_P (14) /* 2.2 */ 
NI_MSG_TOO_LONG_IE_L (1) /* 2.2 */ 

NI_HODGEPODGE_L (15) 

NI_SYNC_GLOBAL_SEND_A (NI_BASE | OxOOCO) 

NI_SYNC_GLOBAL_SEND_L (1) 

NI_INTERRUPT_CLEAR_A (NI_BASE | OxO0c8) 

NI_INTERRUPT_CLEAR_GREEN_A (NI_BASE | 0x0OdO) 

(use same constants as for CAUSE register) 

NI_INTERRUPT_NOW_A (N1_BASE | 0x00d8) 

NI_INTERRUPT_NOW_L (32) 

NI_SCAN_START_A (NI_BASE | 0x0OeO) 

NI_SCAN_START_L (1) 

NI_BAD_ADDRESS_A (NI_BASE | 0x00e8) 

NI BAD ADDRESS L (32) 
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CMOS_signal — asynchronous event handlers on the nodes 
Syntax: 

#include <cmsys/cm_signal.h> 

(*CMOS_signal(sig, func, mask))() 

int sig; 

void (*func) 0; 

int mask; 

Description: 

CMOS_s ignal allows code on the nodes to specify software handlers for certain asynchro¬ 
nous events. It is the responsibility of the user to ensure that the signal handler does not 
change the state of the node in any way that will disrupt execution of the interrupted code. 

A node program can specify that the arrival of Data Network messages with a certain set 
of tags will generate an interrupt. The program specifies the message handler and the set 
of tags with a call to CMOS_s ignal 0 with sig = sigmsg, * func set to the address of 
the user-written handler function, and mask set to a bit mask specifying which tags will 
interrupt. (Bit 0 corresponds to tag 0, bit 1 corresponds to tag 1, and so forth.) Currently, 
tags 0 to 3 are reserved for user messages. Bits 4 and up are reserved for system messages, 
and may not be used or referenced by user code. 

The context of the node except for the floating point context and the global registers g5, 
%g6 , and %g7 is saved before the user message handler is called. Thus, use of floating-point 
instructions in the user message handler will cause unpredictable errors in the interrupted 
code. Also, the network state of the CM is not altered before entering the user message 
handler. Thus, the message(s) that produced the interrupt will still be in the receiving FIFO 
when the user message handler is invoked. It is the responsibility of the user message 
handler to empty these messages. 
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Return Values: 

CMOS_signal () returns the previous action on success. On failure, it returns -1 and sets 
errno to indicate the error. 

Errors: 

CMOS_signal {) will fail and no action will take place if one of the following occurs: 

EINVAL sig was not a valid signal number. 

Notes: 

The handler routine can be declared: 

void handler() 

The routine is not passed any parameters relating to the received message. The user mes¬ 
sage handler must read the NI registers to determine such details as the tag of the message 
and whether the message has arrived via the left or right Data Network interface, etc. 

Message interrupts are disabled while user code is in a user message handler. Thus, user 
message handlers need not be reentrant. However, the message handler should not enable 
interrupts (via a call to CMOS_signal {).) If it does, the results are unpredictable. Also, 
note that if the user code anticipates a series of interrupting messages, the arrival of the first 
message can be used to invoke the message handler and the r emaining messages can be 
received via polling within the handler, thus saving the overhead of an interrupt for all but 
the first message. Message interrupts are disabled by a call to CMOS_signal () with func 
set to CM_SIG_XGN. The mask argument is ignored. (Note that all user tag interrupts are 
disabled by this call.) 
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NI Accessor Examples 




Here axe some examples of macros that C programmers can use to access the 
registers and fields of the NI. In most cases, these macros take as arguments the 
register and field constants defined previously in this manual. 


F.1 Reading and Writing Registers 

The simplest NI register operations involve reading and writing the value of a 
register, typically with one of three types of values: unsigned, float, and double. 
The macros below provide a simple register reading/writing interface. 

#define ni_register(type,reg) *((type *) (reg)) 

tdefine ni_read_reg(reg) ni_register(unsigned, reg) 

#define ni_read_reg_f(reg) ni_register(float, reg) 

#define ni_read_reg_d(reg) ni_register(double, reg) 

#define ni_set_register(type,reg,value) 

ni_register(type, reg) - ((type) (value)) 
#define ni_write_reg(reg) 

ni_set_register(unsigned, reg, value) 
#define ni_write_reg_f (reg) 

ni_set_register(float, reg, value) 
#define ni_write_reg_d(reg) 

ni_set_register(double, reg, value) 

In these examples the reg argument is the address constant of the appropriate 
registrar, and the value argument is the word, float, or double to be written. 
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F.2 Reading and Writing Subfields 

Often, you’ll want to read or write the value of a register subfield. Here’s a set 
of macros that efficiently extract a field from a register. (Note that the field argu¬ 
ment in these examples is the name of the field constant without the _p or _L 
suffixes — these are added automatically by the macros themselves.) 

/* mask for values that will fit into the given field */ 
#define ni_mask_field_values (field_length) \ 

(-(-0 << field_length)) 

/* mask that extracts a field from the register */ 
tdefine ni_mask_field (position, length) \ 

(ni_mask_field__values (length) << position) 

/* right-shift register value, mask out the field */ 
#define ni_get_field(register_val, pos, len) \ 

((register_val >> pos) & ni_mask_field_values(len)) 

#define ni_read_field(register, pos, len) \ 

ni_get_field(ni_read_reg(register), pos, len) 

And here’s a set of macros that efficiently modify the value of a register field: 

/* mask that is ANDed with register to change field */ 
#define ni_new_value_mask(pos, len, new_value) \ 

-((new_value * ni_mask_field_values(len)) << pos) 

/* Logical AND register with mask that changes field */ 
tdefine ni_set_field(xeg_val, pos, len, new_value) \ 
(reg_val & ni_new_value_mask(pos, len, new_value)) 

tdefine ni_write_field(reg, pos, len, new_value) \ 
ni_write_reg(register, \ 

ni_set_field(ni_read_reg(reg), pos, len, new_value)) 

You may also want to simply set or clear an arbitrary set of register bits: 

tdefine ni_set_bits_in_register(reg, bitmask) \ 
ni_write_reg(reg, ni_read_reg(reg) | (bitmask)) 

tdefine ni_clear_bits_in_register(reg, bitmask)\ 
ni_write_reg(reg, ni_read_reg(reg) & -(bitmask)) 
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F.3 Constructing Send-First Addresses 

The only other major set of programming tools that you might need are macros 
that construct a send_f irst address for a given interface. For example: 

#define ni_send_first_a(interface,auxiliary_data) \ 
((unsigned *) ( NIJBASE + \ 

(interface << SF_FIFO_OFFSET [ \ 

auxiliary_data << AUXILIARY_START_P))) 

#define ni_send_first(interface,auxiliary_data,value) \ 
ni_write_reg(ni_send_first_a(interface,auxiliary_data), \ 
value) 


Data Network Send-First Macros 

Here’s a set of macros that constructs the send__f irst addresses for the three 
Data Network interfaces: 

#define ni_xdr_auxiliary_data(mode,tag,length) \ 

( mode << NI_DR_SEND_AUXILIARY_ADDRESS_MODE_P | \ 
tag << NI_DR_SEND_AUXILIARY_TAG_P | \ 

length << ni_dr_send_auxiliary_length_p ) 

#define ni_dr_send_first(mode, tag, length, value) \ 
ni_send_first(DATA_ROUTER_FIFO, \ 

ni_xdr_auxiliary_data(mode,tag,length), \ 
value) 

#define ni_ldr_send_first(mode, tag, length, value) \ 
ni_send_first(LEFT_DR_FIFO, \ 

ni_xdr_auxiliary_data(mode,tag,length), \ 
value) 

#define ni_rdr_send_first(mode, tag, length, value) \ 
ni_send_first(RIGHT_DR_FIFO, \ 

ni_xdr_auxiliary_data(mode,tag,length), \ 
value) 
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Broadcast Interface Send-First Macros 

Here’s a set of macros that constructs the send_f irst addresses for the two 
broadcast interfaces: 

#define ni_xbc_auxiliary_data(length) \ 

( length << NI_BC_SEND_AUXILIARY_LENGTH_P ) 

#define ni_bc_send_first(length, value) \ 
ni_send_first(USER_BC_FIFO, \ 

ni_xbc_auxiliary_data(length), \ 
value) 

#define ni_sbc_send_first(length, value) \ 
ni_send_f irst (SUPERVISOR_BC_FIFO, \ 

ni_xbc_auxiliary_data(length), \ 
value) 


Combine Interface Send-First Macros 


Finally, here’s a set of macros that constructs the send_first addresses for the 
combine interface: 


#define ni_com_auxiliary_data(pattern,combiner,length) \ 
( pattern << NI_COM_SEND_AUXILIARY_PATTERN_P | \ 
combiner << NI_COMJSEND_AUXILIARY_COMBINER_P 1 \ 
length << NI_C0M_SEND_AUXILIARY_LE3SIGTH_P ) 


#def ine ni__bc_send_f irst (pattern, combiner, length, value) \ 
ni_send_f ir s t(COMBINE_FIFO, \ 

ni_com_auxiliary_data(pattern,combiner,\ 

length) \ 


value) 
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Sample NI Programs 


This appendix contains a series of NI programs that test all the programming examples 
shown in the chapters of this manual. For each program, only the PM and node code files 
are given. The interface file for each program is identical to that given for the sample pro¬ 
gram in Chapter 7, and these test programs #include the same utils. h file as is used 
in Chapter 7. 

As of Version 7.1.3 of the CM system software, CMOST, there are on-line copies of the 
sample programs presented here. Depending on where your system administrator has 
stored the CM software, these files may be located under the pathname /usr/cm/src/ 
ni-examples. Check with your system administrator for help in locating these files. 

Important: You should view the examples presented here as merely a cookbook of pos¬ 
sible ideas, not a hard-and-fast ratebook on network protocol. These examples are written 
for clarity, not efficiency, and your own individual application should be your guide as to 
how to rearrange the code fragments presented here, and how best to trim them for speed. 


G.1 Data Network Test 

This program presents examples of a number of different kinds of Data Network 
operations, including 

■ sending/receiving messages limited by the length of the network queues 

■ sendin g and receiving unlimi ted-length messages 

■ using interrupt-driven message retrieval 

■ sending and receiving by the LDR and RDR simultaneously 
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Filename: LDR_test. c 

/* LDR test pxogxain - PM program */ 

#include <cm/cmna.h> 

#include "utils.h" 

#define LONGJFACTOR 5 

void main () { 

int input, result, high_node; 

pxintf("\nLDR test program, by William R. Swanson,\n"); 
pxintf("Thinking Machines Corporation — 2/3/92.\n\n "); 

/* Enable broadcast sending */ 

CMNA_participate_in(NX_BC_SEND_ENABLE); 

/* Abstain from broadcast reception and combine sending */ 
save_and_set_abstain_f lags (1,1,0,0) ; 

/* Start node programs running */ 
node_main(); 

/* Get a value from the user and send it to the nodes. */ 
printf("This CM-5 partition has %d nodes. \n", 
CMNA_partition_size); 

printf ("Please type an integer to send to the nodes: ") ; 
scanf("%d", fcinput); 

PM_s end_t o_NODE(0, input); 

printf("Sent value %d to node 0 ...\n",input); 

/* Wait for the nodes to finish juggling numbers */ 
PM_NODE_synch(); 

/* Get value from high node */ 
highjaode = CMNA_partition_size - 1; 
result = PM_get_from_NODE(high_node); 
printf("Short send:\n"); 

printf("Received value %d (should be %d) from node %d.\n", 
result, input+MAX_BROADCAST_MSG_WORDS-1,high_node); 
result • PM_get_from_NODE(high_node); 
printf("Long send:\n"); 

printf("Received value %d (should be %d) from node %d.\n", 
result, input+(MAX_BROADCAST_MSG_WORDS* 

LONG_FACTOR)-1, high_node); 
result = PM_get_from_NODE(high_node); 
printf("Interrupt-driven send:\n"); 

printf("Received value %d (should be %d) from node %d.\n", 
result, input+MAX_BROADCAST_MSG_WORDS-l,high_node); 
result = PM_get_from NODE(O); 
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printf("Dual-network send:\n"); 

printf("Received value %d (should be %d) from node %d.\n 
result, MAX_BROADCAST_MSG_WORDS, 0); 
restore_abstain_flags(); 


Filename: LDR_test. node. c 

/* LDR test program - node program */ 

#define NI_ROUTER_DONE_P NI_ROUTER_DONE_COMPLETE_P 
#include <cm/cmna.h> 

#include <cmsys/cm_signal.h> 

#include "utils.h" 

#define LONG_FACTOR 5 

/* Send/Receive functions limited by length restriction */ 
int LDR_send (dest_address, message, length, tag) 
unsigned dest_address, tag; 
int ‘message; 
int length; 

{ 

int i; 

CMNA_ldr_send_first(tag, length, dest_address); 
while (length—) CMNA_ldr_send_word(*message++); 
return (SEND_OK(CMNA_ldr_status())); } 

int tag_limit=0; 

int LDR_receive (message, length) 
int ‘message; 
int length; 

{ 

int i, tag = 9 99 ; 

/* Skip messages currently assigned as interrupts */ 
while (tag>tag_limit) { 

if (RECEIVE_OK(CMNA_ldr_ status())) 

tag = RECEIVE_TAG(CMNA_ldr_StatUS{)); 

} 

while (length—) 

*message++ = CMNA_ldr_receive_word(); 
return (tag); 
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/* Send/Receive function with no length restriction */ 
LDR_send_receive_msg(dest_address, message, length, tag, dest) 
unsigned dest_address, tag; 
int *message, *dest; 
int length; 

{ 

int packet_size=MAX_ROUTER_MSG_WORDS-1; 
int send_size, receive_size; 
int offset, source_offset=0, dest_offset; 
int words_to_send=length, words_received=0 ,- 
int count,' rec_tag, status; 

while ((words_received < length) || (words_to_send)) { 

/* First try to receive a packet */ 
status=CMNA_ldr_status(); 
if (words_received<length && 

RECEIVE_OK(status) && 

RECEIVE_TAG(status) <= tag_limit) { 
dest_offset * CMNA_ldr_receive_word() ; 

receive_size - RECEIVE_LENGTH_LEFT(CMNA_ldr_status()); 
for (count=0; count<receive_size; count++) 

dest(dest_offset++] = CMNA_ldr_receive_word(); 
words_received += receive_size; 

} 

/* Now try sending a packet */ 
if (words_to_send) { 

send__size = ( (words_to__send < packet_size) ? 

words_to_send : packet_size); 

do { 

CMNA__ldr_send_f irst (tag, send_size + 1, dest_address) ; 
/* Send offset to indicate part of message being sent 

*/ 

CMNA_ldr_send_word(source_offset); 
offset»source_offset; 

for (count=0; count<send_size; count++) 
CMNA_ldr_send_word(message[offset++]); 

} while (! SEND_OK (CMNA__ldr_status () ) ) ; 
source_offset=offset; 
words_to_send -= send_size; 

} 

} 

} 
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/* Message-receiving handler for interrupt-driven LDR test */ 

int interrupt_done=0; 

int interrupt_expect_length; 

int interrupt_receive[MAX_BROADCAST_MSG_WORDS]; 

void LDR_receive_handler () 

{ 

int temp=tag_limit; 
tag_limit*3; 

LDR_receive(interrupt_receive, interrupt_expect_length) ; 
tag_l imi t=t emp ; 
int er r up t_done=1; 

} 

/* Send/Receive functions using LDR and RDR in tandem */ 
void LDR_RDR_send (dest_address, message, length, tag) 
unsigned dest_address, tag; 
int *message, length; 

{ 

int i ; 

CMNA_ldr_send_first(tag, length, dest_address); 
CMNA_rdr_send_first(tag, length, dest_address); 
for (i=0; iclength; i++) { 

CMNA_ldr_send_word(message[i]); 

CMNA_rdr_send_word(message[i]); 

} 

} 

int LDR_RDR_receive (message, length) 
int *message, length; 

{ 

int i, ldr_value, rdr_value, length_received_ok=0; 
while {!RECEIVE_OK(CMNA_ldr_status()) || 

!RECEIVE_OK(CMNA_r dr_s tatus())) {} 

for (i=0; i<length; i++) { 

ldr_value=CMNA_ldr_receive_word () ; 
r dr_value=CMNA_r dr__r e c e i ve_wo r d () ; 
if (ldr_value==rdr_value) { 
message[i]=ldr_value; 
length_received_ok++; 

} 

} 

return(length_received_ok); 

} 
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/* Combine “network-done" Function */ 
void network_done_synch() 

{ 

CMNA_com_send_first(ASSERT_ROUTER_DONE,SCAN_ROUTER_DONE,1,0) 
while (!DR_ROUTER_DONE(CMNA_dr_s tatus())) {}; 

I 

/* Tool to ensure there's nothing in the receive queues */ 

/* Not used here, but you may find it handy */ 
void LDR_empty_network() { 

int status, length, i; 

while (status=CMNA_ldr_status(), RECEIVE_OK(status)5 
if (RECEIVE_TAG(status) <= tag_limit) { 
length = RECEIVE_LENGTH(status); 
for (i=0; iclength; i++) 

(void) CMNA_ldr_receive_word(); 

} 

} 

void CMPE_node_main () { 

int value*0, i, length=MAX_BROADCAST_MSG_WORDS; 
int 1ong_l ength=1ength* LONG_FACTOR; 
int next_node, mirror_node; 
int received__ok; 

int send(MAX_BROADCAST_MSG_WORDS *LONG_FACTOR), 
receive[MAX_BROADCAST_MSGJWORDS], 

long_receive[MAX_BROADCAST_MSG_WORDS*LONG_FACTOR], 
dual_receive[MAX_BROADCAST_MSG_WORDS]; 

/* signal interrupts for non-zero tag values */ 

CMOS_signal( SIGMSG , LDR_receive_handler , 14 ); 

CMNAjpar ticipat e_in(NI_BC_SEND_ENABLE); 

save_and_set_abstain_flags(0,0,0,0); 

/* All nodes get the value sent by the PM... */ 

All_NODES_get_fr om_PM(&value); 

for( i=0; i<long_length; i++) { 
send[i]=value+i; 
long_receive[i]*-999; 

} 

for( i*0; iclength; i++) { 
receive[i]=-999; 
interrupt_receive[i]*-999; 
dual_receive[i]*-999 ; 

} 
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/* Calculate some useful addresses */ 

next_node * (CMNA_self_address + 1) % CMNA_partition_size 
mirror_node = (CMNA_partition_size-l) - CMNA_self_address 

/* Do an ordinary, length-limited send */ 

LDR_send(next_node, send, length, 0); 
network_done_synch(); 

LDR_receive(receive, length); 
network_done_synch(); 

/* Do an unlimited-length send */ 

LDR_send_receive_msg(mirror_node, send, 

long_length, 0, long_receive); 

network_done_synch(); 

/* Do an interrupt-driven send with a tag of 3*/ 
interrupt_expect_length=length; 

LDR_send(next_node, send, length,3); 
while (!interrupt_done) {} 
network_done_synch(); 

/* Send via both LDR and RDR, and check results */ 
LDR_RDR_send (mirror_node, send, length, 0); 
network_done_synch(); 

received_ok=LDR_RDR_receive (dual_receive, length); 

/* Signal to PM that answer is ready */ 

PM_NODE_synch(); 

/* Send check values back to PM */ 

NODE_send_to_PM(receive[length-1] ); 
NODE_send_to_PM(long_receive [long_length-l] ) ; 
NODE_send_to_PM(interrupt_receive[length-1]); 
NODE_send_to_PM(received_ok); 

restore_abstain_flags(); 

} 
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G.2 Data Network Doubleword Messages Test 

This program demonstrates the use of doubleword read and write operations for 
Data Network transmissions: 

Filename: dbl_test. c 

/* Double-word ops test program - PM program */ 

#include <cm/cmna.h> 

#include "utils.h" 

#define LONG_FACTOR 5 

void main () { 

int input, result, high_node; 

printf("\nDouble-word test program, by w. R. Swanson,\n"); 
printf("Thinking Machines Corporation — 2/3/92.\n\n"); 

/* Enable broadcast sending */ 

CMNA_participate_in(NI_BC_SEND_ENABLE); 

/* Abstain from broadcast reception and combine sending */ 
save_and_set_abstain_flags(1,1,0,0); 

/* Start node programs running */ 
node_main(); 

/* Get a value from the user and send it to the nodes. */ 
printf("This CM-5 partition has %d nodes.\n", 

CMNA_partition_size); 

printf<"Please type an integer to send to the nodes: "); 
scanf("%d", &input); 

PM_send_to_NODE(0, input); 

printf("Sent value %d to node 0 ...\n",input); 

/* Wait for the nodes to finish juggling numbers */ 
PM_NODE__synch () ; 

/* Get value from high node */ 
high_node = CMNA_jpartition_size - 1; 
result = PM_get_from_NODE(high_node); 
printf ("Long send using double-word ops:\n"); 
printf<"Received value %d (should be %d) from node %d.\n", 
r esult, input+(MAX_BROADCAST_MSG_WORDS * 

LONG_FACTOR)-1, high_node); 
restore_abstain_flags(); 

} 
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Filename: dbl_test.node.c 

/* Double-word ops test program - PM program */ 

#include <cm/cmna.h> 

#include <cmsys/cm_signal.h> 

#include "utils.h" 
ttdefine LONG_FACTOR 5 
int tag_limit = 3; 

/* Send/Receive function using double-words */ 
LDR_send__receive_msg_double (dest_address, message, 

length, tag, dest) 
unsigned dest_address, tag; 
int *message, *dest; 
int length; 

{ 

int packet_size; 
double *dbl; 

int send_size, send_size2, receive_size, receive_size2; 
int offset, source_offset=0, dest_offset; 
int words_to_send=length, words_received=0; 
int count, rec__tag, status; 

if ((int)message & 3) 

CMPN_j?anic("Error: Message array not double-word aligned!") 
if ((int)dest & 3) 

CMPN_j?anic("Error: Dest array not double-word aligned!"); 

packet_size = (MAX_R0UTER_MSG_W0RDS-1) & ~1; 

while ((words_received < length) || (words_to_send)) { 

/* First try to receive a packet */ 
status=CMNA_ldr_status(); 
if (words_received<length && 

RECEIVE_OK(status) && 

RECEIVE_TAG(status) <= tag_limit) { 
dest_offset = CMNA_ldr_receive_word(); 

receive_size = RECEIVE_LENGTH_LEFT(CMNA_ldr_status ()); 
printf("received offset %d, size %d.\n", 
dest_offset, receive_size); 

for (count=0; count<(receive_size>>l); count++) { 
dbl = (double *)(&dest[dest_offset++]); 
dest offset++; 
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*dbl = CMNA_ldr_receive_double(); 
dbl++; 

} 

if (ieceive_size & l) /* If word left over */ 
dest[dest_offset++] = CMNA_ldr_receive_word(); 
words_received += receive_size; 

} 

/* Now try sending a packet */ 
if (words_to_send) { 

send_size = ((words_to_send < packet_size) ? 

words_to_send : packet_size); 
send_size2 = send_size >> 1; 
do { 

CMNA_ldr_send_first(tag, send_size + 1, dest_address); 
CMNA_ldr_send_word(source_offset); 
offset-source_offset; 

/* Send as many doubles as possible */ 
for (count=0; count<send_size2; count++){ 
dbl = (double *)(^message[offset++]); 
offset++; 

CMNA_ldr_send_double(*dbl++); 

} 

if (send_size & 1) /* If a word is left over */ 
CMNA_ldr_send_word(message[offset++]); 

} while (!SEND_OK(CMNA_ldr_status())); 
printf("sent offset %d, size %d.\n", 
source_offset, send_size); 
source_offset-offset; 
words_to_send -= send_size; 

} 

} 

} 

/* Combine "network-done" Function */ 
void network_done_synch() 

{ 

CMNA_com_send_first(ASSERT_ROUTER_DONE,SCAN_ROUTER_DONE,1,0 ); 
while (!DR_ROUTER_DONE(CMNA_dr_status())) {}; 

} 

void CMPE_node_main () { 

int value-0, i; 

int leng th=MAX_BROADCAST_MSG_WORDS * LONG_FACTOR; 
int mirror node; 
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/* These variables MUST be double-word aligned! */ 
double temp_dalign_send; 

int s end[MAX_BROADCAST_MSG_WORDS * LONG_FACTOR]; 
double temp_dalign_rec; 

int receive[MAXJBROADCAST_MSG_WORDS*LONG_FACTOR]; 

CMNA_participate_in (NI_BC_SEND_ENABLE) ; 
save_and_set_abstain_flags(0,0,0,0); 

/* All nodes get the value sent by the PM... */ 

Al l_NODES_ge t_f r om_PM {Svalue) ; 

for( i=0; i<length; i++) { 
send[i]=value+i; 
receive[i]=-999; 

} 

mirror_node = (CMNA_partition_size-l) - CMNA_self_address 

/* Do an unlimited-length send using double-word ops */ 
LDR_send_receive_msg_double(mirror_node, send, 

length, 0, receive); 

network_done_synch(); 

/* Signal to PM that answer is ready */ 

PM__NODE_synch () ; 

/* Send check value back to PM */ 

NODE_send_to_PM(receive[length-1]); 

restore_abstain_flags(); 


G.3 Broadcast Interface Test 

This program presents a simple test of broadcasting: 
Filename: BC_test. c 

/* Broadcast examples program - PM program */ 
iinclude <cm/cmna.h> 

#include "utils.h" 
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void main () { 

int input, result, high_node; 

printf{"\nBroadcast test program, by W. R. Swanson,\n"); 
printf("Thinking Machines Corporation — 2/1/92.\n\n"); 

/* Enable broadcast sending */ 

CMNA_participate_in (NI_BC__SEND_ENABLE) ; 

/* Abstain from broadcast reception and combine sending */ 
save_and_set_abstain_flags(1,1,0,0); 

/* Start node programs running */ 
node_main() ; 

/* Get a value from the user and send it to the nodes. */ 
printf("This CM-5 partition has %d nodes.\n", 
CMNA_partition_size); 

printf("Please type an integer to send to the nodes: "); 
scanf("%d", &input); 

PM_send_to_NODE(0, input); 

printf("Sent value %d to node 0 ...\n",input); 

/* Wait for the nodes to finish juggling numbers */ 
PM_NODE_synch(); 

/* Get value from high node */ 
high_node = CMNA_partition_size - 1; 
result = PM_get_from_NODE(high_node); 

printf("Received value %d (should be %d) from node %d.\n", 
result, input+MAX_BROADCAST_MSG__WORDS-l,high_node); 

restore_abstain_flags(); 

} 

Filename: BC_test. node. c 

/* Broadcast examples program - node program */ 

#include <cm/cmna.h> 

#include "utils.h" 

int BC_send(message, length) 
int *message, length; 

{ 

int i ; 

CMNA__bc_send_f irst (length—, *message++) ; 

for (i=0; i<length; i++) CMNA_bc_send_word(*message++); 

return(SEND__OK(CMNA_bc_status 0 ) ) ; 

I 
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int BC_receive(message, length) 
int *message, length; 

{ 

int i ; 

for(i=0; i<length; i++) { 

while(!RECEIVE_OK(CMNA_bc_status())) {} 

message[i] = CMNA_bc_receive_word(); 

} .. 

return(length); 

} 

void CMPE_node_main 0 { 

int value=0, i, length=MAX_BROADCAST_MSG_WORDS; 
int send[MAX_BROADCAST_MSG_WORDS], 

receive[MAX_BROADCAST_MSG_WORDS]; 
int status, rec_length; 

CMNA_par ticipate_in (NI_BC_SEND_ENABLE) ; 
save_and_set_abstain_flags(0,0,0,0); 

/* Node 0 gets the value sent by the PM... */ 
NODE_get_f r om_PM(&value) ; 

for( i=0; i<length; i++) { 

send[i]=value+i; 
receive[i]=-999; 

} 

if (CMNA_self_address==0) { 
status=0; 

while(!status) status = BC_send(send, length); 

} 

rec_length = BC_receive(receive); 

/* Signal to PM that answer is ready */ 
PM_NODE_synch(); 

/* Send value from high-order node back to PM */ 
NODE_send_to_PM(receive[length-1]); 

restore_abstain_flags(); 

} 
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G.4 Combine interface Test 

This program presents examples of a number of different kinds of combine oper¬ 
ations, including 

* sc anning messages, with and without segments 

■ reduction messages 

■ network-done messages 

Filename: cOM_test. c 

/* Combine examples program - PM program */ 

#include <cm/cmna.h> 

#include "utils.h" 

void main () { 

int input, result,. segment_size, high_node, i, expected; 
printf("\nCombine test program, by W. R. Swanson,\n"); 
printf("Thinking Machines Corporation — 2/1/92.\n\n"); 

/* Enable broadcast sending */ 

CMNA_par ticipate_in(NI_BC_SEND_ENABLE); 

/* Abstain from broadcast reception and combine sending */ 
/* Abstain from combine reception, too, for a while... */ 
save_and_set_abstain_flags(1,1,1,0); 

/* Start node programs running */ 
node__main () ; 

/* Get a value from the user and send it to the nodes. */ 
printf("This CM-5 partition has %d nodes.\n", 

CMNA_par tition_size); 

printf("Please type a positive integer: "); 
scanf("%d", &input); 

high_node = CMNA_partition_size-l; 

PM_send_to_NODE(high_node, input); 

printf("Sent value %d to node %d...\n", input, high_node); 

/* Wait for the nodes to finish juggling numbers */ 
PM_NODE_synch(); 

/* Turn combine reception back on */ 
CMNA_write_rec_abstain_flag(com_control_reg, 0); 
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/* Get check values */ 
result = PM_get_f rom_NODE(0) ; 

printf("Received value %d (should be %d) from node %d.\n 
result, (input+MAX_BROADCAST_MSG_WORDS-l), 0) ; 
result = PM_get_from_NODE(high_node); 

printf("Received value %d (should be %d) from node %d.\n 
result, (input*high_node), high_node); 
s egment_si z e = PM_ge t_f r om_NODE(0) ; 
result = PM_get_from_NODE(0); 

printf("Received value %d (should be %d) from node %d.\n 
result, (input+MAX_BROADCAST_MSG_WORDS-l) 

* (segment_size-l), 0); 
result = PM_ge t_f r om_N0DE(0) ; 

printf("Network done for node 0 got %d (should be %d).\n 
result, high_node); 
result = PM_ge t_f r om_NODE(0) ; 

printf("Scanning counted %d nodes (should be %d).\n", 
result, CMNA_partition_size); 

/* Make sure all results are in */ 

PM_NODE_synch(); 

restore_abstain_flags() ; 

} 

Filename: C0M_test. node. c 

/* Combine examples program - node program */ 

#define NI_ROUTER_DONE_P NI_ROUTER_DONE_COMPLETE_P 
#include <cm/cmna,h> 

#include "utils.h" 

int COM_send(combiner, pattern, message, length) 
int ‘message, combiner, pattern, length; 

{ 

int i, start, step; 

/* For max scans, send high-order word(s) first */ 
if (combiner==MAX_SCAN) { start=length-1; step=-l; } 
else { start=0; step=l; } 

CMNA_com_send_first(combiner, pattern, length, 

message(start]); 
for (i=1; i<length; i+ + ) 

CMNA_com_send_word(message[(start+=step)]); 
return(SEND_OK(CMNA_com_s tatus())); 

} 
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int COM_receive(combiner, message) 
int *message; 

{ 

int i, length, start, step; 

while(iRECEIVE_OK(CMNA_com_status())) {} 

length=RECEIVE_LENGTH(CMNA_com_status()); 

/* For max scans, send high-order word(s) first */ 
if (combiner==MAX_SCAN) { start-length-1; step=-l; } 
else { start-0; step=l; } 
for(i=0; i<length; i++) { 

message[start] = CMNA_com_receive_word(); 
start+=step; 

} 

return(length); 


int COM_scan(combiner, pattern, message, length, result) 
int *message, *result, combiner, pattern, length; 

{ 

int status-0, rec_length; 
while (!status) status = 

COM_send(combiner, pattern, message, length); 
rec_length - COM_receive(combiner, result); 
return(rec_length); 

} 

void CMPE_node_main () { 

int value-0, i, length=MAX_BROADCAST_MSG_WORDS; 
int send[MAX_BROADCAST_MSG_WORDS], 
result[MAX_BROADCAST_MSG_WORDS], 
seg_result[MAX_BROADCAST_MSG_WORDS]; 
int rec__length, segment_size, high_node; 
int one, node_count; 

int message, network_done_msg, next_processor; 

CMNA_par ticipate_in(NI_BC_SEND_ENABLE) ; 
save_and_set_abstain_flags{0,0,0,0); 

/* Make sure segmenting is turned off to begin with */ 
CMNA_set_segment_start (0) ; 
high_node = CMNA_partition_size - l; 

/* High node gets the value sent by the PM... */ 
NODE_ge t_f r om_PM (&value) ; 
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/* Fill send array based on supplied value */ 
for( i=0; i<length; i++) { 

send[i]=((CMNA_self_address==high_node) ? value+i : 0); 
result[i]=-999; 
seg_result[i]=-999; 

} 

/* Do a max scan to distribute send values to all nodes */ 
rec_length = COM_scan(MAX_SCAN, SCAN_BACKWARD, send, 

length, send); 

/* Scan overwrites high node — put back original value */ 
if (CMNA_self_address»=high_node) 

for(i=0; i<length; i++) send[i] = value+i; 

/* Do an add scan to make different values */ 
rec_length = COM_scan (ADD_SCAN, SCAN_FORWARD, send, 

length, result); 

/* Do a backwards segmented reduction */ 
segment_size= <CMNA_partition_size<5 ? 

CMNA_partition_size : 5 ); 

CMNA_set_segment_start ( ( (CMNA_self_address % segment_sizel 
== segment_size-l)); 

rec_length = COM_scan(MAX_SCAN, SCAN_BACKWARD, result, 

length, seg_result); 
CMNA_set_segment_start (0) ; 

/* Try network-done feature */ 
me s s age=CMNA_s e1f_addr ess; 
network_done_msg=0; 

next_j3rocessor = (CMNA_self_address+l) 

% CMNA_partition_size; 

CMNA_ldr_send_first (0, l^ext^processor) ; 

CMNA_ldr_send_word(message); 

COM_send(ASSERT_ROUTER_DONE, SCAN_ROUTER_DONE, 
&network_done_msg, 1); 

while (!DR_ROUTER_DONE(CMNA_dr_s tatus{))) {}; 

while (!RECEIVE_OK(CMNA_ldr_status())) {}; 

message=CMNA_ldr_receive_word(); 
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/* Use reduction to do a processor "roll-call" */ 
one=1; 

node_count=-999; 

rec_length = COM_scan(ADD_SCAN, SCAN_REDUCE, 

tone, 1, &node_count); 

/* Signal .to PM that answers are ready */ 
PM__NODE_synch () ; 

/* Send check values back to PM */ 
NODE_send_to_PM{send[length-1]); 

NODE_send_to_PM(result[0]); 
NODE_send_to_PM(segment_size) ; 

NODE_send_to_PM(seg_result[length-1]); 
NODE_send_to_PM (message) ; 

NODE_send_to_PM (node_count) ; 

/* Make sure all results have been received */ 
PM_NODE_synch(); 

restore_abstain_flags() ; 

> 


G.5 Global Network Test 

This program presents a quick example of asynchronous and synchronous global 
interface operations: 

Filename: GLOBAL_test. c 

/* Global network test program - node program */ 

#include <cm/cmna.h> 

#include "utils.h" 

void main () { 

int value; 

printf("\nGlobal test program, by William R. Swanson,\n") 
printf("Thinking Machines Corporation — 2/6/92.\n\n"); 

/* Enable broadcast sending */ 

CMNA_participate_in(NI_BC_SEND_ENABLE); 
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/* Abstain from broadcast reception and combine sending */ 
save_and_set_abstain_flags( 1 ,1,0,0); 

printf("This CM-5 partition has %d nodes.\n", 
CMNA_partition_size); 

/* Start node programs running */ 
printf("Starting node programs ...\n") ; 
node_main(); 

/* Test asynchronous global network */ 
CMNA_or_global_async_bit(0); 

PM_NODE_synch(); 

value = CMNA_global_async_read(); 

printf("Received async bit %d (should be 0).\n", value); 
restore_abstain_flags(); 

} 

Filename: GLOBAL_test. node. c 

/* Global network test program - node program */ 

#include <cm/cmna.h> 
ttinclude "utils.h" 

void CMPE_node_main () { 

int value; 

CMNA_par ticipate_in (NI_BC_SEND_ENABLE) ; 
save_and_set_abstain_flags(0,0,0,0); 

CMNA_or_global_async_bit (0) ; 

/* Signal to PM that answer is ready */ 

PM_NODE_synch(); 

value = CMNA_global_async_read(); 
if (value) 

printf("Error; node got non-zero global value."); 

restore_abstain_flags(); 

} 
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To access the NI constants described in this document, you must #±nclude the 
header file cm/cmna . h: 

#include <cm/cmna.h> 

This file #includes many other header files that provide access to Nl constants, 
register macros, and accessor functions. These constants, macros, and functions 
are collectively refaxed to as CMNA (CM Network Accessors), and can serve as 
a basis for your own NI accessor code. 

Note: The functions and macros in CMNA are designed to be very generic in 
operation. As such, they are much less efficient than the special-purpose macros 
and functions you’ll probably write on your own. Nevertheless, you can use the 
operations defined in CMNA as a jumping-off point for your own code, to help 
you understand what needs to be done to get your code to run correctly. 


H.1 What Is CMNA? 

There are two main parts to CMNA: 

■ The NI Interface — Constants and macros used to manipulate NI registers. 

■ CnC (“C-and-C”) — C functions that perform NI operations such as 

reading and writing messages of arbitrary length. 

The CMNA header files define the NI interface explicitly, in terms of register 
accessor macros and constants. The header files also provide C prototypes for the 
CnC functions, which are part of the CMOST operating system code. 
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H.2 CMNA Header Files 

The following header files are part of CMNA: 

/usr/include/ 

cm/cmna. h — Main CMNA header file, 

cmsys/cmna .h — CMNA user header file, 

cmsy s /cmna_sup . h — CMNA supervisor header file. 

cmsys/ni_interf ace. h — Main NI interface header file. 
cmsys/ni_macros . h — NI macro definitions. 

cn»sys/ni_constants. h — NI register/flag constant definitions, 
cmsys/ni_def lnes. h — Low-level NI constant definitions. 

The following diagram shows the relationship among the header files that make 
up CMNA: 



Figure 21. Relationship between CMNA and NI header files. 
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H.2.1 The Main CMNA Header File: cm/cmna.h 

This single file #includes all the header files that are needed to define CMNA. 
However, it contains virtually no definitions of its own. It simply #includes 
either of the two header files cmsys/cmna. h or cmsys/cmna_sup. h, accord¬ 
ing to which NI register area (user or supervisor) the #±ncludeirig code needs. 

Implementation Note: At present, cmsys/cmna_sup. h is only iincluded 
for diagnostic code (that is, code that defines the symbol cmdiag). 


H.2.2 The User Header File: cmsys/cmna.h 

This file #includes the NI constant and macro files described below, and also 
defines a number of useful C mask constants and C macros that are used in 
CMNA However, the constants and macros defined here are only sufficient for 
the needs of CMNA and are not by any means a complete set. (See the descrip¬ 
tion of the n±_constants. h, and ni_def ines files below.) 


H.2.3 The Supervisor Header File: cmsys/cmna_sup.h 

This file modifies a few key constant definitions so that any absolute memory 
address constants defined in the other header files will refer to the NI supervisor 
area, rather than the NI user area. It then #includes cmsys/cmna. h, so it has 
much the same effect as that header file. 

Note: The cmsys/cmna_sup. h file is only of use to programmers with legal 
access to the NI supervisor area. Including this file does not in itself grant access 
to the NT’s supervisor area; it simply redefines many CMNA constants to have 
address values that are only legal for supervisor code. 
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H.2.4 The NI interface Header File: nijnterface.h 

This file defines the NI accessor interface. It #includes the file 
ni_constants .h, and defines a number of basic NI register macros that are 
used by CMNA. It then #includes ni_macros. h to define the remainder of the 
CMNA macros. 

This file also defines a number of NI register constants that are suitable for use 
in C code. (That is, constants that have been cast as (unsigned *) values. See 
the description of ni_constants .h and ni_defines .h below.) 


H.2.5 The NI Macros Header File: ni_macros.h 

This file defines a number of C macros that perform sterotypical NI operations 
such as sending and receiving messages via a specific network interface. 


H.2.6 The NI Constants Header Files: 
ni_constants„h, ni_defines.h 

These files define a number of register constants and masks that are used by 
CMNA. In particular, ni_constants. c includes definitions of constants speci¬ 
fying the absolute memory address for each of the NTs registers. The file 
ni_def ines. h defines hundreds of constants that give the size and offset of the 
register fields of the NI. These two sets of constants provide a complete interface 
for NI operations written in assembly code. Appendix D provides a complete list 
of these constants, grouped by register and category. 

Note For C Programmers: The register address constants are unsigned pointer 
values. To use them in C code, you must first cast them to type (unsigned *). 
For example: 

unsigned *ni_dr_status = ((unsigned *) NI_DR_STATUS); 

If you don’t perform this casting step, the C compiler by default treats the 
constants as signed integers, possibly causing your code to fail. Many of these 
constants are recast in just this fashion in the header file ni_inter f ace . c, so 
you may be able to just use those constants without having to do any recasting 
yourself. 
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H.3 CMNA Functions 

Below are listed the basic functions provided by the CMNA library aside from 
those that directly access the NI chip (which are described elsewhere in this 
manual). 

Important: The functions defined here are designed for usability, not perfor¬ 
mance. In actual production applications, you will want to write your own 
routines to obtain the highest communications performance possible. Use the 
routines described below as examples of how you might write an NI accessor 
function, not as hard and fast examples of how such a function should be written. 


H.3.1 CMNA Version 

CMNA_ver aion() 

Returns: char * 


H.3.2 Activity Functions 

CMNA_absta±n_from (activity) 

CMN A p articipate In (activity) 

CMNA_sup_abstain_from {activity) 
CMNA_sup_jpart±cipate_in (activity) 
int activity; 

Return: int 

/* valid activity to participate in or abstain from 


participating in: */ 


#define 

NI_REDUCE_RECEIVE 

(1) 

#define 

NI_BC_RECEIVE 

(2) 

#define 

NI_COMBINE 

(4) 

#define 

NX_SYNC_GLOBAL 

(8) 

#define 

NI_SBC_RECEIVE 

(16) 

#define 

NI_BC_SEND_ENABLE 

(32) 
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H.3.3 DR Interface Functions 

CMNA_dr__msg_to_receive () 

Returns: int 

CMNA_dr_send_f if o_amount (dest_proc, source_base, 

word_length,tag) 

unsigned int dest_proc; 
void *source_base; 
int word_length; 
unsigned int tag; 

Returns: int 

CMNA_dr_send_f if o_amount_phys±cal (dest_proc, source__base, 

word_length, tag) 

void *source_base; 
int word_length; 
unsigned dest_proc; 
unsigned tag; 

Returns: int 

CMNA_dr_send_jmsg (des t_pr oc,source_base,word_length,tag) 

unsigned int dest_proc; 

void *source_base; 

int word_length; 

unsigned int tag; 

CMNA drsend ms gp hya i cal (dest_proc,source_base, 

word_length,tag) 

void *source__base; 
int word_length; 
unsigned dest_proc; 
unsigned tag; 

CMNA_dr_s tatus() 

Returns: unsigned 


H.3.4 LDR Interface Functions 

CMNA_ldr__receive (base) 
void *base; 

Returns: int 
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CMNA_ldr_receive_msg (base, word_length) 

void *base; 

int word_length; 

CMNA_ldr_send_fif o_amount (dest_proc,source_base, 

word_length,tag) 

unsigned int dest_proc; 
void *source_base; 
int word_length; 
unsigned int tag; 

Returns: int 

CMNA_ldr_aend_£ ifo_amount_physical (dest_proc,source_base, 

word_length,tag) 

void *source_base; 
int word_length; 
unsigned dest_proc; 
unsigned tag; 

Returns: int 

CMNA_ldr_send_msg (dest_jproc, source_base, word_length, tag) 

unsigned int dest_j?roc; 

void *source_base; 

int word_length; 

unsigned int tag; 

CMNA ldr sendms gp hysical (dest_proc,source_base, 

word_length,tag) 

void *source_base; 
int word_length; 
unsigned dest_proc; 
unsigned tag; 


H.3.5 RDR Interface Functions 

CMNA_rdr_receive (base) 
void *base; 

Returns: int 

CMNA__r dr_r e c e i ve_ms g (base, wor d_length) 

void *base; 

int word_length; 
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CMNA_r dr_send_f i f o_amoun t(dest_pxoc,souxce_base, 

woxd_length,tag) 

unsigned int dest_proc; 
void *source_base; 
int woxd_length; 
unsigned int tag; 

Returns: int . 

CMNA rdr send£ifoamoun t p hysical (dest_pxoc,souxce_base, 

woxd_length,tag) 

void *source_base; 
int woxd_length; 
unsigned dest_proc; 
unsigned tag; 

Retuxns: int 

CMNA_rdr_send_msg (de s t_pioc,souxce_base,woxd_length,tag) 

unsigned int dest_pxoc; 

void *souxce_base; 

int woxd_length; 

unsigned int tag; 

CMNA_rdr_send_msg_physical (dest_pxoc,souxce_base, 

woxd_length,tag) 

void *souxce_base; 
int word_length; 
unsigned dest_pxoc; 
unsigned tag; 


H.3.6 BC Interface Functions 

CMNA_bc_r e ad_doub1e() 

Returns: double 

CMNA_bc_read_f loat 0 

Returns: float 

CMNA_bc_read_int() 

Returns: int 

CMNA_bc_read_uint() 

Returns: unsigned 
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CMNA_bc__receive (msg,length) 
void *msg; 

■unsigned int length; 

CMNA_bc_receive_participation() 

Returns: int 

CMNA_bc_send_and_receive_msg (msg,result,length) 
void *msg; 
void *result; 
int length; 

CMNA_b c_s end_f1fo_amount (msg,length) 
void *msg; 
int length; 

CMNA_bc_send_msg (msg,length) 
void *msg; 
int length; 

CMNA_bc_wait_f or_receive_ok {) 

Returns: int 

CMNA_bc_write_doub1e (data) 
double data; 

CMNA_b c_write_float (data) 
float data; 

CMNA_bc_write_int (data) 
int data; 

CMNA_bc_wr ± te_uint (data) 
unsigned int data; 


H.3.7 SBC Interface Functions 

CMNA_sbc_double (data) 
double data; 

CMNA_sbc__f loat (data) 
double data; 
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■a«»»r^w'3ww^: 


;$£ £' y ; *- 


CMNA_sbc_int(data) 
int data; 

CMNA_sbc_receive (msg, length) 
void *msg; 
unsigned length; 

CMNA_sbc_send(msg,length) 
void *msg; 
int length; 

CMNA_sbc_send_msg(msg,length) 
void *msg; 
int length; 

Returns: int 

CMNA_sbc_uint(data) 
unsigned int data; 

CMNA_sbc_wait_f or_receive_ok () 

Returns: int 

CMNA_sup_dr_send_packet_to_scalar 

(source_base, word_length, tag) 
void *source_base; 
int word_length; 
unsigned tag; 

Returns: int 

CMNA_sup_ldr_send_packet_to_scalar 

(source_base,word_length,tag) 
void *source_base; 
int word__length; 
unsigned tag; 

Returns; int 

CMNA sup rdr send packet to scalar 

(source_base,word_length,tag) 
void *source_base; 
int word_length; 
unsigned tag; 

Returns: int 
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H.3.8 COM Interface Functions 

CMNA__com (combiner,pattern,data,length,result) 

int combiner; 

int pattern; 

void *data; 

int length; 

void *result; 

CMNA com participation() 

Returns: int 

CMNA_com_receive (result) 
unsigned int *result; 

CMNA_com_send (combiner,pattern,data,word_length) 

int combiner; 

int pattern; 

void Mata; 

int word_length; 

CMNAreducere c p articipation() 

Returns: int 


H.3.9 Global Interface Functions 

CMNA_global_async (value) 
unsigned int value; 

Returns: int 

CMNA_global_sync (value) 
unsigned int value; 

Returns: int 

CMNA_gl oba l_sync_r e ad () 

Returns: int 

CMNA_g 1 oba l_sync_x e ad_when_r e ady () 

Returns: int 

CMNA_global_sync_par ticipation () 

Returns: int 
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NI Chip Version 2.2 Changes 


This appendix presents a summary of the additions and changes made to the NI 
chip as of the most recent chip version (2.2), and indicates where they are 
described in the main body of this manual. 


1.1 Long Data Network Messages 

The Data Network now has the capability to send long messages. These mes¬ 
sages, sent by a special register interface, have a length in data bytes that is much 
longer than the limit imposed on Data Network messages in Version 1.0. (Cur¬ 
rently, the long message length limit is 18 words of data.) 

Several new Data Network registers are introduced to support this feature: 

ni_dr/ldr/rdr_send_f irst_long 
nl_dr/ldr/rdr_stafus_long 
ni_longes t_dr_me a a age 

The long message register interface is described in detail in Chapter 3. Section 
3.4.2 in particular describes how to send long messages. 
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1.2 New Data Network Status Interface 

Registers have been added to allow more convenient access to the status informa¬ 
tion of the Data Network’s message FIFOs, and to allow “popping” of messages 
from the Data Network receive FIFOs at the same time: 

ni_dr/ldr/rdr_status__all 
ni__ldr /rdr_8 tatus_pop 

These registers are described in Section 3.5. 


1.3 New Data Network Tag Interrupt Interface 

The mechanism for detecting and signaling interrupts based on the tag values of 
Data Network messages has been changed to allow more precise control over the 
selection of user and supervisor tag values. The new mechanism is described in 
detail in Section 3.5.4. 


1.4 Non-Compatible Change to Broadcast Interface 

The ni_rec_length_left field in the ni_bc_status and ni_sbc_status 
registers has been expanded from 4 bits to 7 bits in length to handle the change 
in the maximum broadcast message length. This means that software written for 
earlier versions of the NI chip may not execute properly; if only the first four bits 
of this field are extracted, the software cannot determine whether the value thus 
obtained is correct. 

The basic fix for this problem is to have your code extract 7 bits rather than 4 
for this field. If your code uses the predefined NI constants, you should substitute 
the new length constant ni__bc_rec_i.enqth_left_lono_l for all references to 
the rec_length_lef t field. 
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1.5 New Interrupts 


A number of new interrupts have been added as of Version 2.1: 


rdone complete 

(Orange) — 

message too 

long 

(Yellow) — 

dperr 


(Green) — 

sfifo empty 


(Green) — 

ldr rec tag 


(Green) — 

rdx rec tag 


(Green) — 

ldr user rec 

tag 

(Green) — 

rdr user rec 

tag 

(Green) — 


These interrupts are described in Chapter 


Completion of network-done operation 
Data Network message length error 
Error signaled by CM-5 vector units 
Data Network send fifo empty 
LDR supervisor message tag interrupt 
RDR supervisor message tag interrupt 
LDR supervisor message tag interrupt 
RDR supervisor message tag interrupt 

5, and in Appendix B. 


1.6 New Data Network Interrupt Enable Flags 

The following flags have been added to the Data Network private registers to 
allow enabling and disabling of the corresponding interrupts: 

ni_sf if o_goes_empty_ie 
n±_rdone_complete__ie 

These flags are described in Section 3.8. 


1.7 New Bus Error Conditions 

The following bus error conditions now exist, in connection with the long mes¬ 
sage feature of the Data Network: 

■ writing a message to any of the Data Network’s send_first registers 
with a length value that is greater than either five words or the value of the 
register n±_longest_dr_message, whichever is less 

■ writing a message to any of the Data Network’s send_f irst_long reg¬ 
isters with a length value that is greater than the value of the register 
ni_longest_dr_message 
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1.8 Disabling Bus Errors 

The flag ni_d±sable_bus_error in the hodgepodge register, when set, 
causes the NI to signal bus errors as a yellow interrupt, bad_memory_access. 
(See Section 5.1.1.) 


1.9 Manually Triggering Interrupts 

Interrupts can be triggered artificially by writing to the new registers 
ni_±nterrupt_set and ni_±nterrupt_set_gzeen (See Section 5.3). 


1.10 Global Interface Context-Switching 

A supervisor method for context-switching and then restoring the state of the 
synchronous global interface is described in Section 4.3.2. 


1.11 New Hodgepodge Register Fields 

The following fields have been added to the “hodgepodge” register to support 
various new NI features: 

ni_d±sable_bus_error (See Section 5.1.1.) 

ni_ldr_rec_tag_ie (See Section 3.5.4.) 

n l_r dr _r e c_t ag_i e ” 

ni_ldr_user_rec_tag_ie ” 

ni_rdr_user_rec_tag_ie ” 

ni_msg_too_long_ie (See Section 3.2.2.) 
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This index lists the register names and fields, programming constants, functions, and macros referred 
to within this document Bold page numbers indicate a def ining reference or important description. 


A 

ADD_SCAN 

combine combiner constant, 75,152 
combiner constant, 77,186 

ASSERT_ROUTER_DONE 

combine combiner constant, 75,152 
combiner constant, 79,186 
auxiliar.y_stab.t _3 send-first field offset 
constant, 21,149 

B 

bad memory access 
bus error, 97,178 
Yellow interrupt, 96, 98, 100,170 
bad relative address. Yellow interrupt, 
41, 96,100,172 

be interrupt green. Green interrupt, 

97,101,104,173 

be Interrupt orange, Orange interrupt, 

96.100. 104. 169 

be Interrupt red, Red interrupt, 96, 

100, 104, 168 

be Interrupt yellow, Yellow interrupt, 

96. 100. 104. 170 

be or com collision, Yellow interrupt, 
73, 96,100, 104, 171 
be rec ok, Green interrupt, 30, 174 
bc_control_reg, constant, 69,185 


c 

CMNA_bc_r e ceive_type (), macro, 68,184 
CMKA_bc_send_fiirst (), macro, 67,184 
CMNA_bc_s end_f ir s t_doub 1 e (), macro, 
67,184 

CMNA_bc_send_type {), macro, 67,184 
CMNA_bc_status (), macro, 68,184 
CMNA_com_rec e ive_type (), macro, 76, 
186 

CMNA_com_send_f irst () , macro, 74,186 
CMNA_com_send_f irst_double (), macro, 
~ 74,186 

CMNA_com_send_type (), macro, 74,186 
CMNA_com_atatus £), macro, 76,187 
am&_dinterface_T e ce ive_type () , macro, 

44.182 

CMNA_dmre//ace_send_first (), macro, 

42.182 ~ 

cmSK_dinterface_n end_£ i r s t_doub 1 e, 
macro, 42,182 

casA_dinteiface_BBnd_£ irst_double_lon 
g, macro, 43 

CUNA_dinterface _aBnd_£ ir s t_l ong (), 
macro, 43 

cmtA_dinterface_Bend_type, macro, 42,182 
cmiA_dinteTface_Bta.tuB (), macro, 46,183 
CMNA_dr_send_status (), macro, 46,183 
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CMNA_global_async_read (), macro, 92, 

” 189 

CMNA global sync complete (), macro, 

” 90,188 

CMNA_global_sync_rec (), macro, 90,188 
CbmA_interface_xec&Lve_type, macro, 23 
ctsSK_interface_BBoA_t irst (), macro, 22 
CKJHK_interface_B end_f i r s t_doubl e, 
macro, 22 

CMNA Interface aen d p acketto 

_scalar (), system function, 122 
cmtAjnterface_Bend_type(), macro, 23 
dSSK_interface_B fcatus (), macro, 26 
CMNA_ldx_status (), macro, 46,183 . 
CMNA_oz_global_async_bit (), macro, 
92,189 

Q4NA_or_gl°bal_8ync_bit (), macro, 90, 
188 

CMMA partlclpate ln O 
system call, 141 
system function, 70,124 
CMNA_partition_size, variable, 40,181 
CMNA_zdz_status (), macro, 46,183 
CMNA_read_abstain_£lag(), macro, 28, 

” 181 

CMNA_segment_stazt (), macro, 78,187 
CMNA_sel£_address, variable, 40,181 
CMNA_set_segment_start (), macro, 78, 
187 

CMNA_write_abstain_f lag 0, macro, 28, 
181 

CMOS_signal (), system call, 51,199 
emu erzoz, Red interrupt, 96,99,167 
cn checksum ezzoz. Red interrupt, 96, 

99,166 

cn hard ezzoz, Red interrupt, 96,99,167 
com abstain changed, Yellow interrupt, 
82,96,100,170 

com zee empty, Green interrupt, 83,97, 
101,175 

com zee ok, Green interrupt, 30,97,101, 
174 

com_contzol_reg, constant, 82,188 
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combine_fifo, interface number constant, 
22,150 

combine_overfi»ow (), macro, 79,187 

D 

data_router_fifo, interface number 
constant, 22,150 
dp ezzoz, Green interrupt, 176 
dpezz, Green interrupt, 97,101 
dz checksum ezzoz, Red interrupt, 96, 
99,166 

dz count negative, Yellow interrupt, 52, 
96,100,171 

dz zee all £all down, Green interrupt, 
55,97,101,174 

dz zee ok. Green interrupt, 30,97,101, 
174 

dz zee tag, Green interrupt, 49,97,101, 

174 

dr_receive_state (), macro, 53,183 
dr_rooter_done (), macro, 54, 80, 183 
dr_send_s tate (), macro, 53, 183 

G 

global zee, Green interrupt, 93,97,101, 

175 

I 

inteznal fault, Red interrupt, 96,99, 
166 

L 

ldz zee ok, Green interrupt, 30,97,101, 
174 

ldz zee tag, Green interrupt, 97 
ldz tag, Green interrupt, 177 
ldz usez zee ok. Green interrupt, 101 
ldz usez zee tag. Green interrupt, 97 
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ldr user tag, Green interrupt, 177 
left_dr_fifo, interface number constant, 
22,150 

M 

MAX_BROAE>CA3T_MSG_WORDS, constant, 66, 

67,149,185 

max_combxne_msg_words, constant, 73, 
149,187 

max_router_msg_words, constant, 41, 43, 
149,184 

max_sbc_msg_words, constant, 66, 67,149 

MAX_SCAN 

combine combiner constant, 75,152 
combiner constant, 77,186 
me error, Red interrupt, 96,99,167 
message too long, Yellow interrupt, 96, 
100,172 

N 

ni_al 1_£ al l_down_enab 1 e, flag, 54, 55, 
154 

ni_all_fall_down_±e, flag, 54,55,154 
ni_async_global, register, 89,92,146, 
160,189 

ni_async_sup_global, register, 89,93, 
146,160* 

ni_bad_addxess, register, 114,146,164 
ni_bad_addxess_low, field, 114,164 
nl_bad_address_type, field, 114,164 
ni_base, constant, 12,21,149 
nl_bc_..., register. See uX_binterface_... 
ni_bc_control, register, 148,158,185 
nl_bc_prlvate, register, 148,158 
ni_bc_recv, register, 148,184 
n±_bc_send, register, 148,184 
KI_BC_SEND_AOXILIARX_LENGTH_g field 

“offset! 67,151 

nl_bc_send__f irst, register, 148,184 
nl_bc_status, register, 148,157,185 
■ai._binterface_Gout.xoX, register, 64, 69 
nL_binterface_px ivate, register, 64,69 
ui_bintejface_xBcv, register, 64,67 
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u±_binterface_aeud, register, 64,67 
n±_binterface_Ben<i_t irst, register, 64,67 
uX_binterface_atutMB, register, 64, 68 
n±_cause_bad_memory_access, flag, 161 
n±_cause_bad_relat±ve_address, flag, 
161 

n±_cause_bc_interrupt_gxeen, flag, 
161 

n±_cause_bc_interrupt_orange, flag, 
161 

ni_oause_bc_interrupt_red, flag, 161 
ni_cause_bc_interrupt_yellow, flag, 
161 

ni_caus e_bc_or_o om_co 11 i s i on, flag, 
161 

nl__cause_bc_rec_ok, flag, 161 
ni_cause_cmu_error, flag, 161 
n±_cause__cn_checksum_erroE, flag, 161 
ni_cause_cn_hard_error, flag, 161 
n±_cause_com_abstain_changad, flag, 
161 

nl_cause_com_rec_empty, flag, 161 
ni__cause_com_rec_ok, flag, 161 
n±_cause__dperr, flag, 161 
nl_o aus e_dr_checksum_e r r or, flag, 161 
ni_cause_dr_oount_negatlve, flag, 161 
nl_cause_dr_rec_all_fall_down, flag, 
161 ”* ™ 

ni_cause_dx_rec_ok, flag, 161 
nl_cause_dr_rec_tag, flag, 161 
ni_cause_global_rec, flag, 161 
n±_cause_internal_f aul t, flag, 161 
ni_eause_ldr_rec_ok, flag, 161 
n ±_c aus e_l dr_r e c_t ag, flag, 161 
n±_eause_ldr_user_rec_tag, flag, 161 
ni_cause_mc_error, flag, 161 
ni_cause_message_too_long, flag, 161 
n±_cause_rdone_complete, flag, 161 
ni_cause_rdr_rec_ok, flag, 161 
n l_c aus e_r dr_r e c_tag, flag, 161 
nl_cause_rdr_user_rec_tag, flag, 161 
ni_cause_sbc_rec_ok, flag, 161 
nl_cause_scan_overf low, flag, 161 
ni_cause_sf ifo_empty, flag, 161 
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ni_cause_Bupervisor_global_rec, flag, 
161 

ni_cause_sync_global_rec, flag, 161 
ni_cause_timer_interrupt, flag, 161 
ni_chunk_size, register, 110,146 
ni_chunk_table_address, register, 112, 
146 

ni_chunk_table_data, register, 112,146 
nl_clearJbad_memory_aceess, flag, 162 
ni__dear_bad_relative_address, flag, 
162 

ni_cleax_bc_interrupt_green, flag, 

162 

nl_clear_bc_interrupt_orange, flag, 
162 

ni__clear_bc_Interrupt_red, flag, 162 
ni_clear_bc_interrupt_yellow, flag, 
162 

n±_clear_bc_or_com_collision, flag, 
162 

ni_clear__bc_rec_o3c, flag, 162 
ni_clear_cmu_error, flag, 162 
ni_clear_cn_checks-um_error, flag, 162 
ni_clear_cn_hard_error, flag, 162 
ni_clear__com_abstaln_changed, flag, 
162 

ni_clear_com_rec_enrpty, flag, 162 
ni_clear_com_rec_ok, flag, 162 
ni_clear_dperr, flag, 162 
ni_e 1 e ar_dr_che cksum_e r r or, flag, 162 
ni_clear_dr_count_negative, flag, 162 
ni_e 1 e ar__dr_r e c_a 1 l_f a 1 l_down, flag, 
162 

ni_clear_dr_rec_ok, flag, 162 
n i__c 1 a ar_dr_r e c_t ag, flag, 162 
ni_clear_global_rec, flag, 162 
ni_clear__internal_fault, flag, 162 
ni_c 1 e ar_l dr_r e c_o k, flag, 162 
ni_clear_ldr_rec_tag, flag, 162 
ni_elear_ldr_user_rec_tag, flag, 162 
ni_clear_jnc_error, flag, 162 
nl_cleaz_nesaage_too_long, flag, 162 
ni_cleaz_rdon.e_coinplefce, flag, 162 
ni_clear__rdr_rec_pk, flag, 162 
ni__clear_zdz_rec_tag, flag, 162 
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ni_clear_zdr__user_rec_tag, flag, 162 
ni_cleaz_sbc_rec_ok, flag, 162 
ni_clear_scan_overf low, flag, 162 
ni_clear_sfi£o_empty, flag, 162 
niclearsupervlso rq lobalrec. flag, 
162 

ni_clear_syne_global_rec, flag, 162 
ai_clear__timer_interrupt, flag, 162 
ni_cn_stop_send, flag, 107, 116,163 
ni_rec_abstain, flag 
of a network, 27, 28 
of broadcast interface, 69 
of combine interface, 81 
ni_com__control, register, 72, 81,148,160, 
188 

ni_com_f lush_send, register, 113,146 
n.1 com pr ivate, register, 72, 83,148,159 
ni_oojn_reo_eu 5 >ty_ie, flag, 83 
in ni_com_private register, 159 
ni_com_recv, register, 72, 76,148, 186 
n±_com_scan._overf low, flag, 76, 78 
in ni_com_status register, 159 
of combine interface, 187 
ni_com_scan_overflow_ie, flag, 79, 83 
in nl com private register, 159 
nl_ooin_send, register, 72, 73, 83, 148, 186 

NI_COM_SEND_AtrXILXARY_COMB XNER_P, 

field offset, 74,151 

NI_COM_SEND_AUXILXARY_LENGTH_R field 

offset, 74,151 

NI_COM_SEND_AUXILIARY_PATTERN_P, 

field offeet, 74,151 

ni_coin_send_coinblnez, field, 83, 84 
in ni_comjprivate register, 159 
ni_com_send_f ir b t, register, 72, 73,148, 
186 

ni_ooin_send_length, field, 83, 84 
in ni_com_private register, 159 
ni_com_send_pattern, field, 83, 84 
in ni_com_private register, 159 
ni_com_send_start, flag, 83, 84 
in nl_oom_private register, 159 
nl_coin_status, register, 72, 76, 78, 148, 
159, 187 

ni_conf iguration, register, 115,146 
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ni_conf iguratlon_complete, flag, 107, 
115,163 

ni_count_mask, register, 38, 51, 80, 146 
ni_dinterface_px ivate, register, 36, 54 
ni_dinterface_xec_ta.g, field, 183 
nl_dinterface_xecv, register, 36, 44, 182 
ni_dinterface_aend, register, 36, 42, 182 
ni_dinterface_aend_£ list, register, 36, 42, 
182 

ni_dinterface_aend_£ Irs t_long, register, 
36,43 

n±_dinterface_ata.tna, register, 36, 45, 80 
ni_dinterface_ata.tuB {), register, 183 
nl_dinterface_Bta.tuB_a.ll, register, 36, 46 
n±_dinterface_a tatus_long, register, 36 
nl_dinterface_ata.tuB_pop, register, 36, 47 
n±_d±sable_bus_error, flag, 98,107, 163 
ni_dr_.... See nl _dinterface_... 
ni_dr_meaaage_count, register, 38, 51, 

54, 80,146 

ni_dr_pr ivate, register, 147,154 
ni_dr_rec_al l_f all_down, flag, 54, 55 
in n±_dr_pr ivate register, 154 
in ai_ldx_private register, 156 
in ni_rdx_jpr ivate register, 157 
ni_dr_rec_state, field, 45, 53,153,154 
ai_dr_rec_tag, field, 45 

in ai_dr_a tatus register, 153 
in ai_ldr_s tatus register, 155 
in ni_ldr_status_long register, 155 
in n±_rdr_status register, 156 
in ai_rdr_status_loag register, 156 
in ni_dr_status_long register, 154 
ni_dr_aend, register, 147 

NI_DR_SEND_AUXXLIARY_ADDRE3 S_MODE_ 

p, offset constant, 42,150 

NI_DR_SEND_AOXXLXARY_LENGTH_g offset 

constant, 42,150 

NI_DR_SEND_At3XILXARY_TAG_g offset 
constant, 42,150 

al_dr_sead_f irst, register, 147 
ni_dr_sead_f ±rst_long, register, 147 
ni_dr_sead_ok, flag, in 

ni_dx_status_all/pop register, 
46,154 


ai_dr_sead_space, field, in 

ni_dr_statua_all/pop register, 
46,154 

ai_dr_aead_state, field, 45, 53,153,154 
ai_dr_s tatus, register, 147,153 
ai_dx_status_all, register, 147 
ni_dr_status_all/pop, register, 154 
ni_dr_status_long, register, 147,154 
ai_f lush_complete, flag, 107,113,163 
ni_global_rec, flag, 92,160,189 
a±_global_rec_ie, flag, 92, 93,107,163 
ni_global_sead, flag, 92,160, 189 
ai_hodgepodge, register, 38, 49, 98,107, 
146,163 

and asynchronous global interface, 89 
and supervisor asynch global interface, 

89 

and synchronous global interface, 89 
asynch global rec interrupt enable flag, 
92, 93 

broadcast interrupt flags, 104 
configuration flag, 115 
flush complete flag, 113 
NI timer interrupt enable flag, 113 
send stop flag, 116 

supervisor rec interrupt enable flag, 93 
synch global rec interrupt enable flag, 89, 
91 

ni_interface_contxol, register, 17,25, 27 
ni._interface_px±vBte, register, 17, 30 
nljnterface_purpose, register naming 
format, 10 

ni_in terfa ce_x e cv, register, 17,23 
n±_interface_aend, register, 17,19 
nl_interface_aend_£ ir at, register, 17,19, 
149 

nl_interface_aend_£ ir st__long, register, 
43, 152 

ni._interface_ata.tMB, register, 17 
n±_interrupt_cause, register, 102,146, 
161 

n±_interrupt_cause_green, register, 

102,146,161 

ni_interrupt_clear, register, 102,146, 
162 


NI Version 2.2 (CM-5E), June 1994 

Copyright © 1994 Thinking Machines Corporation 


247 



NI Programmer’s Handbook 


ni_interrupt_clear_green, register, 
102,146,162 

ni__interrupt_level, register, 103,146, 
163 

ni_interrupt_level_green, field, 103, 
163 

ni_interrupt_level_orange, field, 103, 
163 

ni__interrupt_level_red, field, 103,163 
ni__interrupt_level_yellow, field, 103, 
163 

ni_interrupt_now, register, 113,146 
ni_interrupt_rec_enable, flag, 104, 

107,163 

ni_lnterrupt_eend, register, 104,146 
n.i_interrupt_send_olc, flag, 104, 107, 
163 

ni_lnterrupt_Bet, register, 146,162 
ni__interrupt_set_green, register, 146, 
162 

ni_ldr_.... See n±_dinterface_... 
ni_ldr_prlvate, register, 147,156 
ni_ldr_rec_a 1 1__£ a 1 l_dovm, flag 

in ni_dr_status_all/pop register, 46, 

154 

in ni_ldr_Btatus_all/pop register, 

155 

in ni_rdr_BtatuB_all/pop register, 
157 ~ 

ni_ldr_rec_length, field 

in ni_dr_s tatus_al 1 /pop register, 154 
in nl_ldr_Btatus_all/pop register, 
155 " 

in ni_rdr_status_all/pop register, 
157 

ni_l dx_r e c_l eng th_l ong, field, in 

ni_dr_s tatus_al 1 /pop register, 
46 

ni_ldr_zec_ok, flag 

in ni_dr_status_all/pop register, 46, 

154 

in ni_ldx_status__all/pop register, 

155 ~ 

in n±_rdr_status_all/pop register, 
157 
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ni_ldr_rec_tag, field 

in ni_dr_status_all/pop register, 46, 

154 

in ni_ldr_status_all/pop register, 

155 

in ni_rdr_status_all/pop register, 
157 * 

n±_ldx_rec_tag_ie, flag, 49,107, 163 
ni_ldr_recv, register, 147 
ni_ldx_send, register, 147 
ni_ldz_Bend_f irst, register, 147 
ni_ldr_send_f irst_long, register, 147 
n±_ldr_send_ok, flag, in 

ni_ldr_Btatus_all/pop register, 
155 

ni_ldr_send_space, field, in 

ni_ldr_s tatus_al 1 /pop register, 
155 

ni_ldr_status, register, 147,155 
ni_ldr_status_all, register, 147 
nl_ldr_status_all/pop, register, 155 
nl_ldr__statuB_long, register, 147,155 
nl ldr status pop, register, 147 
ni_ldr_user_rec_tag_ie, flag, 49,107, 
163 

ni_lock, flag 

in nl b e p rivate register, 158 
in nl co m p rivate register, 159 
in ni_dr_pr Ivate register, 154 
in ni_ldx_pr ivate register, 156 
in ni_rdr_private register, 157 
in nl sbe private register, 158 
of a network, 30, 31 
of broadcast interface, 69 
of combine interface, 83 
of Data Networks, 54 

ni_longest_dr_message, register, 38,39, 
146 

ni_message_too_long_ie, flag, 38,107 
ni_msg_too_long_ie, flag, 163 
ni_partition_base, register, 108, 110, 
146 

nl_j>artition_size, register, 108, 109, 
146 

nl_physical_sel£, register, 108,146 
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ni_rdone_complete_ie, flag, 54, 57,154 
ni_rdr_.... See ni._dinterface_... 
ni_rdr_private, register, 147,157 
ni_rdx_rec_all_f all_down, flag 

in nl_dr_s tatus_all /pop register, 46, 
154~ 

in ni_ldr_statua_all/pop register, 

155 

in ni_rdr_status_all/pop register, 
157 ~ 

ni_zdr_rec_length, field 

in ni_dr_s tatus_all /pop register, 154 
in ni_ldr_atatus_all/pop register, 
155 

in n±_rdz_statua_all/pop register, 
157 

ni_rdr_rec_length_long, field, in 

ni_dr_s tatus_al 1 /pop register, 
46 

ni_rdr_rec_ok, flag 

in n±_dz_s tatus_all /pop register, 46, 

154 

in ni_ldr_atatua_all/pop register, 

155 

in ni_r dx_s t atus_al 1 /pop register, 
157 

n±_rdr_rec_tag, field 

in ni_dr_statua_all/pop register, 46, 
154~ 

in ni_ldr_s tatua_al 1/pop register, 
155 

in ni_rdr_status_all/pop register, 
157 

ni_rdr_rec_tag_ie, flag, 49,107,163 
n±_xdr_recv, register, 147 
nl__rdr_send, register, 147 
ni_r dr_aend_f ir a t, register, 147 
ai_rdr_.3end_first_.long, register, 147 
ai_rdr_aend_ok, flag, in 

ni_rdr_status_all/pop register, 
157 

al_rdr_sead_apace, field, in 

ni_rdr_status_all/pop register, 
157 

ni_rdr_8tatus, register, 147,156 
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ni_rdr_status_all, register, 147 
ni_rdr_status_all/pop, register, 157 
ni_rdr_status_long, register, 147,156 
ni_rdr_status_pop, register, 147 
ni_rdr_user_rec_tag_ie, flag, 49,107, 
163 

ni_rec_abstain, flag 

in ni_bc_eontrol register, 158 
in ai_com_control register, 160 
in ni_sbc_control register, 159 
of broadcast interface, 185 
of combine interface, 188 
ai_rec_full, flag 

in ni_bc_private register, 158 
in ni_com_private register, 159 
in ni_dx_private register, 154 
in ni_ldr_private register, 156 
in ni_rdr_pz±vate register, 157 
in ni_sbc_private register, 158 
of a network, 30,32 
of broadcast interface, 69 
of combine interface, 83 
of Data Networks, 54 

ni_rec_intezrupt_maak, register, 38, 49, 
146 

ni_rec_length, field 

in ni_com_s tatua register, 159 
in n±_dr_status register, 153 
in ni_dr_status_long register, 154 
in ni_ldr_statua register, 155 
in ni_ldr_status_long register, 155 
in nl_rdr_s tatua register, 156 
in nl_rdr_status_long register, 156 
of a network, 25,26 
of a network interface, 181 
of combine interface, 76,187 
of Data Networks, 45,183 
ni_zec_length_lef t 
field 

in ni_bc_s tatua register, 157, 238 
in ni_com_s tatua register, 159 
. in ai_dr_a tatua register, 153 
in ni_dr_status_long register, 154 
in ni_ldr_status register, 155 
in ni_ldr_Btatus_long register, 155 
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in ni_rdr__status register, 156 
in ni_rdr_status_long register, 156 
in ni_Bbc_statuB register, 158 
in nl_Bbc_status register, 238 
of a network, 25, 26 
of broadcast interface, 68, 68, 185 
of combine interface, 76, 187 
of Data Networks, 45, 183 
flag, of a network interface, 181 
ni_rec_ok, flag 

in n±_bc_status register, 157 
in n.i_com_statuB register, 159 
in ni_dr_s tatus register, 153 
in ni_dr_statu8_long register, 154 
in ni_ldr_status register, 155 
in ni_ldr_status_long register, 155 
in ni_rdr_s tatus register, 156 
in ni_rdr_status_long register, 156 
in ni_sbe_status register, 158 
of a network, 25, 26 
of a network interface, 181 
of broadcast interface, 68,185 
of combine interface, 76, 187 
of Data Networks, 45, 183 
ni_rec_ok_ie, flag 

in ni_bc_j>rivate register, 158 
in nl drprlvate register, 154 
in n±_ldr_private register, 156 
in nl rdrprivate register, 157 
in ni_sbc_private register, 158 
in ni_com_private register, 159 
of a network, 30, 30 
of broadcast interface, 69 
of combine interface, 83 
of Data Networks, 54 
ni_rec_state, field, of Data Networks, 
183 

nl_rec_stop, flag 

in n±_bc_pr ivate register, 158 
in ni com prlvate register, 159 
in ni_dr__pr ivate register, 154 
in ni_sbc_pr ivate register, 158 
of a network, 30, 31 
of combine interface, 83 
of Data Networks, 54 
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ni_rec_tag, field, of Data Networks, 183 
n±_reduce_rec_abstain, flag, 81,160, 
188 

of combine interface, 27,28 
ni_router_done_coinplete, flag, 45,46, 
54,76,80,153,154 

in ni_ldr_status_all/pop register, 
155 

in ni_rdr_status_all/pop register, 
157 ” 

of Data Networks, 183 
ni_sbc_..., register. See ni_binterface_... 
ni_sbc_control, register, 148,159 
ni_sbc_pr ivate, register, 148,158 
ni_sbc_recv, register, 148 
ni_sbc_send, register, 148 
ni_sbc_send_f ir s t, register, 148 
ni_sbe_status, register, 148,158 
nl_scaa._start, register, 72, 78,146, 187 
ni_8end_empty, flag 

in ni_bc_status register, 157 
in ni_com_status register, 159 
in ni_sbc_status register, 158 
of a network, 25 
of a network interface, 181 
of broadcast interface, 68,185 
of combine interface, 76, 187 
ni_send_enable, flag 

in nl_bc^private register, 158 
in ni_sbc_private register, 158 
of broadcast interface, 69,69 
n.i_send_ok, flag 

for Data Networks, 45 

in ni_bc_status register, 157 

in ni_eom_status register, 159 

in ni_dr_s tatus register, 153 

in n i_dr_B ta tus_l ong register, 154 

in ni_ldr_status register, 155 

in nl_ldr_status_long register, 155 

in nl_rdr__status register, 156 

in ni_rdr_status_long register, 156 

in ni_sbc_B tatus register, 158 

of a network, 25, 25 

of a network interface, 181 

of broadcast interface, 68,185 
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of combine interface, 76,187 
of Data Networks, 183 
n±_sead_space, field 

in n±_bc_s tatus register, 157 
in ni_com_status register, 159 
in ni_dr_atatus register, 153 
in ni_dx_status_long register, 154 
in ni_ldx_status register, 155 
in n±_ldr_status_long register, 155 
in nl_rdr_s tatus register, 156 
in ni_xdx_status_long register, 156 
in n±_sbc_status register, 158 
of a network, 25, 26 
of a network interface, 181 
of broadcast interface, 68,185 
of combine interface, 76,187 
of Data Networks, 45,183 
n±_send_state, field, of Data Networks, 
183 

ni_send_stop, flag, of broadcast interface, 
30,32,69 

nl_sexlal_numbex, register, 116,146 
n±_set_bad_memory_access, flag, 162 
ni_set_bad_xelat±ve_addxess, flag, 

162 

ni_set_bc_intexxupt_gxeen, flag, 162 
n±_set_bc_intexxupt_orange, flag, 162 
ni_set_bc_intexxupt_xed, flag, 162 
n±_set_bc_inteixupt_yellow, flag, 162 
nl_set_be_ox_com_collision, flag, 162 
ni_set_bc_xec__ok, flag, 162 
ni_set_cmu_exxox, flag, 162 
ni_set_en_checksum_exxox, flag, 162 
nl_set_cn_baxd_exxox, flag, 162 
ni_set_com_abstain_changed, flag, 162 
ni_set_com_xec_empty, flag, 162 
nl_s e t_c om_x e c_ok, flag, 162 
ni_set_dpexx, flag, 162 
ni_set_dx_checksunt_exxox, flag, 162 
ni_set_dx_Gount_negative, flag, 162 
u i_s e t_dx_r ec_all_fal l_down, flag, 

162 

ni_set_dx_xec_ok, flag, 162 
ni_set_dx_xec_tag, flag, 162 
nl_set_global_xec, flag, 162 
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ni_se t_intexnal__f aul t, flag, 162 
ni_set_ldx_iec_ok, flag, 162 
nl_set_ldx_xec_tag, flag, 162 
ni_set_ldx_usex_xec_tag, flag, 162 
ni_set_mc_exxox, flag, 162 
ni_Bet_message_too_long, flag, 162 
ni_set_xdone_complete, flag, 162 
ni_set_xdx_xec_ok, flag, 162 
ni_set_xdx_rec_tag, flag, 162 
n±_set_xdx_usex_xec_tag, flag, 162 
ni_set_sbc_xec_ok, flag, 162 
ni_set_scan_ovexflow, flag, 162 
ni_set_sf i£o_en^ty, flag, 162 
ni_set_supexvisox_global_xec, flag, 
162 

ni_set_sync_global_xec, flag, 162 
n±_set_timex_intexxupt, flag, 162 
ni_sf i£o_goes_empty_ie, flag, 54, 57, 
154 

ni_supexvisox_global_xec, flag, 93,160 
n±_supexvisox_global_xec_ie, flag, 93, 
107,163 

ni_supexvisox_global_send, flag, 93, 
160 

ni_sync_global, register, 89, 89,146,160, 
188 

ni_sync_global_absta±n, register, 89, 

90.146.188 

ni_sync_global_complete, flag, 89, 90, 

160.188 

ni_sync_global_xec, flag, 89, 90,160, 

188 

ni_sync_global_xec_ie, flag, 89, 91, 
107,163 

ni_sync_global_send, register, 89, 90, 

146,188 

nl_time, register, 113,146 
ni_timex_ie, flag, 107,113,163 
ni_usex_rec_lntaxxupt_mask, register, 
38, 49,146 

ni_usei_tag_mask, register, 38, 48,146 
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O 

OR_SCAN 

combine combiner constant, 75,152 
combiner constant, 77,186 

P 

physical, flag value constant, 43,150 

R 

xdone complete, Orange interrupt, 96, 
100 

rdr rec ok. Green interrupt, 30,97,101, 
174 

rdr rec tag, Green interrupt, 97 
rdr tag, Green interrupt, 177 
rdr user rec ok. Green interrupt, 101 
rdr user rec tag, Green interrupt, 97 
rdr user tag, Green interrupt, 177 
RECEIVE__LENGTH (), macro, 27,181 
RECEIVE_LENQTH_LEFT (), macro, 27, 181 
RECEXVE_OK (), macro, 27,181 
receive_tag (), macro, 48,183 
relative, flag value constant, 43,150 
right_dr_fifo, interface number 
constant, 22,150 

router done complete, Orange 
interrupt, 57,169 

S 

sbc rec ok, Green interrupt, 30,97,101, 
174 

seem overflow, Green interrupt, 79,97, 
101,176 

SCAN_BACKWARD 

combine pattern constant, 75,151 
pattern constant, 77,186 

SCAN_FORWARD 

combine pattern constant, 75,151 
pattern constant, 77,186 

SCANJREDUCE 

combine pattern constant, 75,152 


pattern constant, 77,186 

S CAN_ROUTER_DONE 

combine pattern constant, 75,152 
pattern constant, 79,186 
send fifo earpty, Green interrupt, 57, 

177 

send_empty (), macro, 27,181 
sendjdk {), macro, 27,181 
s end_s pace (), macro, 27,181 
sf_fifo_offsei send-first field offset 
constant, 21,149 

sf Ifo empty, Green interrupt, 97,101 
sp-pe-stubs, preprocessor, 127 
supervisor global rec, Green 
interrupt, 93,97,101,175 
stoervisorjbcjfifo, interface number 
constant, 22,150 

sync global rec, Green interrupt, 91, 97, 
101,175 

sync global abstain reg. constant, 90, 
188 

T 

timer interrupt. Orange interrupt, 96, 
100,114,168 

u 

UADD_SCAN 

combine combiner constant, 75,152 
combiner constant, 77,186 
oser_bc_fifo, interface number constant, 
~ 22,150 

V 

vo error. Green interrupt, 176 

X 

XORJ3CAN 

combine combiner constant, 75,152 
combiner constant, 77,186 
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This index lists the essential concepts referred to within this document. Bold page numbers indicate a 
de finin g reference or important description. 


absolute address, in chunk table 
translations, 110 
abstain flag, 27 
effect of, 28 

function to set values of, 123 
in control registers, 9 
of broadcast interface, 69 
of combine interface, 28, 81 
for reduction operations, 28 
of global interface, 90 
using efficiently, 138 
using safely, 29 
abstaining 

from a network interface, 27 
from a synchronous global message, 90 
from broadcast interface, 69 
from combine interface, 81 
addition (signed), combine operation, 75 
addition (unsigned), combine operation, 75 
addition scan overflow, 78 
address (node) registers, 108 
address translation, and NI ch unk table, 
108 

addresses 

calculating send_£ixse, 21 
calculating sesd_.flrst_.long, 43 
of registers, 145 

programming constants, 11 
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addressing 

of nodes, 39,119,139 
of partition manager, 119 
of registers, programming constants, 11 
physical. See physical addressing 
relative. See relative addressing 
ali gnme nt of doubleword data, 140 
“All Fall Down interrupt enable” flag, 54, 
55 

‘All Fall Down message” flag, 54,55 
All Fall Down Mode, 55 
address word format, 56 
detecting, 55 
resending, 56 
triggering, 55 

“All Fall Down mode enable” flag, 54,55 
alternate status register, of Data Networks, 
36,46 

asynch global receive interrupt, 93 
“asynch global receive interrupt enable” 
flag, of asynchronous global 
interface, 92,93,107 

asynch supervisor global receive interrupt, 
93 

“asynch supervisor global receive” flag, of 
supervisor asynch global interface, 
93 
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“asynch supervisor global send” flag, of 

supervisor asynch global interface, 
93 

“asynch supervisor global” register, of 

supervisor asynch global interface, 
89,93 

“asynch supervisor receive interrupt 

enable”, of supervisor asynch global 
interface, 93,107 
“asynch global receive” flag, of 

asynchronous global interface, 92 
“asynch global send” flag, of asynchronous 
global interface, 92 

“asynch global” register, of asynchronous 
global interface, 89,92 
asynchronous interface, of global interface, 
88,89 

auxiliary information, 20 
for broadcast messages, 67 
for combine messages, 74 
for Data Network messages, 41,42 
of a network message, 18 

B 

backward scan, combine pattern, 75 
“bad address low” field, 114 
“bad address type” field, 114 
“bad address” register, 114 
base address, of NI memory region, 8 
programming constant, 12,21 
broadcast enabling, 69 
CMOST operation for, 141 
broadcast interface, 3,63,64 
abstaining from, 69 
auxiliary information, 67 
broadcast interrupt interface, 104 
conflicts with combine interface, 140 
enabling, 69 

CMOST operation for, 141 
message format, 66 
message ordering, 66 
messages, 65 
receiving, 67 
registers, 64 


sending, 66 

supervisor broadcast interface, 64 
user broadcast interface, 64 
“broadcast interrupt receive enable” flag, 
104,107 

“broadcast interrupt send ok” flag, 104,107 
“broadcast interrupt send” register, 104 
broadcast interrupts. See interrupts, 
broadcast 

broadcast messages, user and supervisor, 64 
“Bus Error disable” flag, 107 
Bus Errors, 97,178 

and bad address register, 114 
on abstain flag change during global 
message, 90 

on bad memory access, 97,178 
on broadcast interrupt error, 104 
on broadcasting with sending disabled, 

69 

on combine flush error, 113 
on configuration error, 115 
on excessively long messages, 19 
on improper message format, 19 
on network-done message error, 80 
on reading from empty rec FIFO, 26 
on reading/writing undefined addresses, 

7 

on sending with abstain flag set, 28, 90 
on user access to supervisor features, 7 
on user sending message with supervisor 
tag, 48 

on user sending physical mode message, 
43 

C 

casting register constants, for C coding, 11 
chunk address, 109 
chunk position, 110 
“chunk size” register, 110 
chunk sizes, 110 
chunk table, 40,108 
modifying, 112 
size of chunks, 110 
“chunk table address” register, 112 
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“chunk table data” register, 112 
clearing combine send FIFO, 83 
ca_signal. h, header file, 51 
CM-5,2 
networks, 2 
operating system, 5 
partition manager, 4 
partitions, 4 
processing nodes, 3 
programs, 5 

CMMD, software interface, 1 
CMNA 

(CM Network Accessors), 225 
header files, 226 

cmna.h, header file, 11,13,125,225 
code 

for nodes, 5 
for PM, 5 

“combine add-scan overflow” flag, 76,78 
combine flush, 112 

“combine flush complete” flag, 107,113 
“combine flush” register, 113 
combine interface, 3,63,71 
abstaining from, 81 
auxiliary information, 74 
conflicts with broadcast interface, 140 
flushing, 112 
message format, 73 
message ordering, 73 
messages, 73 

network-done messages, 79 
parallel prefix. See scanning 
pipelining, 73 
receiving, 76 
reduction messages, 77 
registers, 72 
scan overflow, 78 
scanning, 77 
sending, 73 
status register, 76 
word order in scans, 77,140 
combine messages, word order in, 140 
combine patterns 
addition (signed), 75 
addition (unsigned), 75 


backward scan, 75 
exclusive OR, 75 
forward scan, 75 
inclusive OR, 75 
ma ximum , 75 
network-done, 75 
reduction, 75 

combiner field, combine interface, legal 
values, 75 

“combiner value” supervisor field, of 
combine interface, 83,84 
combiner values, for combine messages, 77 
communications networks. See CM-5 
networks; networks 
compiling NI programs. See programs 
configuration, partition, 115 
“configuration complete” register, 107,115 
“configuration” register, 115 
conflicts, between broadcast and combine 
interfaces, 140 

Connection Machine CM-5 Technical 
Summary, xix 
constants 

NI base address, 12,21 
programming, 11 
register, address, 11 
register field, position and length, 12 
Control Network, 2,3,63 
See also broadcast interface; combine 
interface; global interface 
disabling, 116 

“Control Network disable” flag, 107,116 
control register, register type, 9 
“control” register 

of a network interface, 17,27 
of broadcast interface, 64,69 
of combine interface, 72,81 
“count mask” register, 38,51,80 
“current” message, in receive FIFO, 25 

D 

Data Network (DR), 2,2,35 
addressing. See addressing 
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All Fall Down mode, 55 
address word format, 56 
detecting, 55 
resending, 56 
triggering, 55 

auxiliary information, 41,42 
chunk table, 108 

interactions between interfaces, 36 

length field, 42 

message format, 41 

message mode bit, 42 

message modes, physical and relative, 40 

message ordering, 39 

message tags, 48 

messages, 38 

auxiliary information, 42 
length field, 42 
mode bit, 42 
tag field, 42 
receiving, 44 
registers, 36 

send FIFO, registers, 42 
sending, 42,43 
tag value of messages, 42 
Data Network interfaces 
Data Network (DR), 36 
left interface (LDR), 2,36 
registers, 36 

See also Data Network 
right interface (RDR), 36 
detecting arrival of messages, 24 
Diagnostic Network, 3 
disabling the Control Network, 116 
discarded messages, 20 
and sen.d__ok flag, 25 
using efficiently, 138 
doubleword data, alignment, 140 
doubleword operations, for reading/writing 
registers, 19 

doubleword operators, 22,136 
“DR length limit” register, 38,39 
“DR network done” flag, 45,46,54, 76 
“DR receive state” field, 45,53 
“DR send state” field, 45,45,46,53 
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E 

examples, on-line, 134 

exclusive OR, combine operation, 75 

executing NI programs. See programs 

F 

fields, register 

See also register fields 
position and length constants, 12 
fields and flags, status. See status register, 
fields and flags 

“flush complete” flag, 107,113 
“flush” register, of combine interface, 113 
flushing, the combine interface, 112 
format of messages, 18,19 
for asynchronous global interface, 92 
for broadcast interface, 66 
for combine interface, 73 
for Data Network, 41 
for supervisor asynch global interface, 93 
for synchronous global interface, 90 
forward scan, combine pattern, 75 

G 

generic network interface, 17 
using effectively, 32 
getting value of status register, 26 
See also status registers 
“global abstain” register, of synchronous 
global interface, 89,90 
global interface, 3,63,88 
asynchronous interface, 91 
supervisor asynch interface, 93 
“global receive” register, of synchronous 
global interface, 89, 89 
“global send” register, of synchronous 
global interface, 89,90 
Green broadcast interrupt, 104 
Green interrupt, 97,101,173 

Green broadcast interrupt, 97,101,104, 
173 

on add scan overflow, 79,97,101,176 


NI Version Z2 (CM-5E), June 1994 
Copyright © 1994 Thinking Machines Corporation 



Concepts Index 




on All Fall Down message receipt, 55,97, 

101.174 

on DP (vector unit) error, 176 
on empty combine receive FIFO, 83,97, 

101.175 

on empty Data Network send FIFO, 57, 
97,101,177 

on interrupting DR message tag, 49,97, 

101.174 

on LDR/RDR tag, 177 
on LDR/RDR user tag, 177 
on message receipt, 30,91, 93,97,101, 

174.175 

on vector unit error, 97,101,176 
“Green interrupt clear” register, 102 
“Green interrupt level” field, 103 


header files 

cm_signal.h, 51 
cmna.h, 11,225 

“hodgepodge” register, 38,49,107 
and asynchronous global interface, 89 
and supervisor asynch global interface, 
89 

and synchronous global interface, 89 
broadcast interrupt flags, 104 
configuration flag, 115 
flush complete flag, 113 
global receive interrupt enable flag, 92, 
93 

NI timer interrupt enable flag, 113 
send stop flag, 116 

supervisor receive interrupt enable flag, 
93 

sync global receive interrupt enable flag, 
89,91 

1 

inclusive OR, combine operation, 75 
interface, register 

of asynchronous global interface, 91 
of broadcast interface, 64 
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of combine interface, 72 
of Data Networks, 36 
of global interface, 89 
of supervisor asynch global interface, 93 
of synchronous global interface, 89 
interface code file. See programs 
“interrupt cause” register, 102 
“interrupt clear” register, 102 
“interrupt level” register, 103 
“interrupt now” register, 113 
interrupts, 13,95,165 
and tag fields, 48 
broadcast, 104 
Bus Errors, 178 

on bad memory access, 178 
Bus Errors, 97 

and bad address register, 114 
on abstain flag change during global 
message, 90 

on bad memory access, 97 
on broadcast interrupt error, 104 
on broadcasting with sending disabled, 
69 

on combine flush error, 113 
on configuration error, 115 
on excessively long messages, 19 
on improper message format, 19 
on network-done message error, 80 
on reading from empty receive FIFO, 
26 

on reading/writing undefined 
addresses, 7 

on sending with abstain flag set, 28,90 
on user access of supervisor features, 

7 

on user sending message with 
supervisor tag, 48 
on user sending physical mode 
message, 43 

cause and clear registers, 102 
classes, 13,95 
detecting and clearing, 102 
Green, 97,101,173 

on add scan overflow, 79 
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on All Fall Down message receipt, 55 
on broadcast interrupt, 104 
on empty receive FIFO, 83 
on interrupting DR message tag, 49 
on message receipt, 30,91,93 
interrupt levels, 103 
Orange, 96,100,168 

on broadcast interrupt, 104 
on NI timer interrupt, 114 
pathways, 98 
recovery, 98,105 
Red, 95, 99,166 
off-chip faults, 99 
on broadcast interrupt, 104 
on-chip faults, 99 
using to retrieve Data Network 
messages, 48 
Yellow, 96,100,169 

on bad relative address, 41 
on broadcast interrupt, 104 
on broadcast/combine collision, 73 
on broadcast/combine conflict, 104 
on Bus Error signaled as interrupt, 98 
on combine/abstain flag error, 82 
on negative message count, 52 
IOR, combine operation, 75 

L 

“LDR supervisor tag interrupt enable” flag, 
49,107 

“LDR user tag interrupt enable” flag, 49, 
107 

left Data Network interface (LDR), 2,35 
length limit 

of network interface FIFOs, 19 
on broadcast interface messages, 66 
length of message 
remaining words, 26 
total (as received), 26 
“lock” flag 

of a network interface, 30,31 
of broadcast interface, 69 
of combine interface, 83 
of Data Network interfaces, 54 
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M 

mapping, relative to physical addresses. 111 
maximum, combine operation, 75 
memory maps 

network interface registers, 18 
node virtual memory, 9 
of broadcast interface registers, 65 
of combine interface registers, 72 
of Data Network registers, 37 
of global interface registers, 88 
memory subsystem, of nodes, 3 
“message count” register, 38,51,54, 80 
message counting, 51 
in network-done operations, 80 
message format 

asynchronous global interface, 92 
broadcast interface, 66 
combine interface, 73 
Data Network, 41 

supervisor asynch global interface, 93 
synchronous global interface, 90 
message ordering, broadcast interface, 66 
message tags, 48 
user/supervisor, 48 

“Message too long interrupt enable” flag, 
38,107 
messages 

broadcast interface, 65 
combine interface, 73 
word order, 140 
Data Network, 38 
detecting arrival of, 24 
discarded, 20 

and send_ok flag, 25 
format, 18 

for asynchronous global interface, 92 
for broadcast interface, 66 
for combine interface, 73 
for Data Network, 41 
for supervisor asynch global interface, 
93 

for synchronous global interface, 90 
from nodes to PM, 121 
from PM to nodes, 120 
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global interface, 88 
length field, for Data Network, 42 
mode bit, for Data Network, 42 
modes, (for Data Network), 40 
network, 18 

receipt order, for Data Network, 39 
receiving, 23 

microprocessor, of processing node, 3 
“middle” Data Network interface, 2 
“middle” Data Network interface 
restrictions, 140 

N 

“network done” flag 
See also “DR network done” flag 
of Data Network, (network-done 
operation), 80 
Network Interface (NI), 2,6 
base address, 8 
constant, 12,21 
chip, 2,6 

interrupts, 13,95,165 

memory region, occupied by registers, 7 

memory regions, physical and virtual, 8 

operation times, 135 

performance hints, 135 

register names, 10 

register types, 9 

registers, 6 

Reset, 14,117 

serial number, 116 

supervisor area, 7 

timer, 113 

user area, 7 

network interfaces, interactions between, 

140 

network-done interrupt enable flag, 54 
network-done 

combine interface operation, 71,79 
combine operation, 75 
message format, 79 
network-done messages, (via combine 
interface), 79 
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networks, 2 

co mm on features, 17 
conflicts between. See broadcast 
network, conflicts; combine 
network, conflicts 
interface, registers, 17 
interface numbering, 21 
interfaces, generic, 17 
messages, 18 

NI. See Network Interface (NI) 

NI programs. See programs 
NI Reset, 117 

“NI timer enable” flag, 107,113 
node, program, 5 
node program. See programs 
nodes. See processing nodes 

O 

off-chip faults, (Red interrupts), 99 
on-line code examples, 134 
on-chip faults, (Red interrupts), 99 
operating system. See CM05 operating 
system 

operation times, of NI, 135 
OR, combine operation, 75 
See also XOR, combine operation 
Orange broadcast interrupt, 104 
Orange interrupt, 96,100,168 
network-done complete, 57,169 
NI timer interrupt, 96,100,114,168 
Orange broadcast interrupt, 96,100,104, 
169 

router-done complete. See Orange 

interrupt, networkj&jdone complete 
“Orange interrupt level” field, 103 
order of words, in scan messages, 77 
overflow, in addition scans, 78 

P 

parallel prefix, combine interface 
operation. See scanning 
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partition 

See also partitions 
size of, variable, 40 

“partition base address” register, 108,110 
partition configuration, 115 
“partition configuration” register, 115 
partition manager (PM), 4 
address of, 40,119 
code, 5 

exchanging data with nodes, 119 
program. See programs 
“partition size” register, 109 
partitioning, by system administrator, 4 
partitions, 4 

configuration, 115 
defined by the NI chunk table, 108 
relative addressing within, (for Data 
Network), 40 
size, 4 

pattern field, combine interface, legal 
values, 75 

pattern values, for combine messages, 77 
performance hints, 135 
physical addressing 
See also addressing 

translation from relative addressing, 109 
physical base address, of NI memory 
region, 8 

“physical self address” register, 108 
pipelining combine operations, 73 
PM program. See programs 
“private” register, 30 

of a network interface, 17,24,30 
of broadcast interface, 64,69 
of combine interface, 72,83 
of Data Network interface, 36,54 
processing node program. See programs 
processing nodes, 2,3 
address registers, 108 
address translation, 108 
addresses of, 39 
registers, 108 
addressing. See addressing 
exchanging data with PM, 119 
internal structure, 3 
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progr amming models, user and OS, 5 
programs 

compiling and executing, 132 
interface code file, 127 
NI, 5,12 

node code file, 126 
PM and node, 4 
PM code file, 125 
structure of, 125 
protocol 

See also messages, format 
for sending messages, 19 

Q 

FIFO register 

of a network interface. See receive FIFO 
register; send FIFO registers 
register type, 9 

R 

“RDR supervisor tag interrupt enable” 
register, 49,107 

“RDR user tag interrupt enable” flag, 49, 
107 

reading a message, 23 
reading registers, using doubleword 
operators, 136 
reading status registers, 26 
“receive abstain” flag 

for broadcast interface, 69 
of a network, 27,28 
of combine interface, 81 
of global interface, 90 
“receive FIFO empty interrupt enable” 
flag, of combine interface, 83 
“receive FIFO full” flag 
of a network, 30,32 
of broadcast interface, 69 
of combine interface, 83 
of Data Networks, 54 
“receive ok interrupt enable” flag 
of a network, 30,30 
of broadcast interface, 69 
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of combine interface, 83 
of Data Networks, 54 
“receive interrupt mask” register, 38, 49 
“receive length left” field 
of a network, 25,26 
of broadcast interface, 68 , 68 
of combine interface, 76 
of Data Networks, 45 
“receive length” field 
of a network, 25,26 
of combine interface, 76 
of Data Networks, 45,46 
“receive ok” flag 

of a network, 24, 25,26 
of broadcast interface, 68 
of combine interface, 76 
of Data Networks, 45,46 
receive FIFO 

network register for, 23 
of a network, 9,18,23 
receive FIFO register, of a network, 23 
“receive state” field, of Data Network, 45, 
53 

“receive stop” flag, of a network, 31 
“receive” register 
of a network, 17,23 
of broadcast interface, 64,67 
of combine interface, 72, 76 
of Data Networks, 36,44 
receiving 

a broadcast interface message, 67 
a combine interface message, 76 
a Data Network message, 44 
a global interface message, 92 
a network message, 18,23 
a network-done message, 80 
a reduction-scan message, 77 
a scan message, 77 
a synchronous global message, 90 
an asynch supervisor global message, 93 
an asynchronous global message, 92 
Red broadcast interrupt, 104 
Red interrupt, 95, 99,166 
off-chip faults, 99 
on cache/MMU error, 96,99,167 


on Control Network checksum failure, 
96, 99,166 

on Control Network hardware failure, 
96, 99,167 

on Data Network checksum failure, 96, 
99,166 

on memory controller error, 96, 99,167 
on NI chip fault, 96, 99,166 
on-chip faults, 99 

Red broadcast interrupt, 96,100,104, 
168 

“Red interrupt level” field, 103 
reduction 

combine interface operation, 71,77 
See also scanning 
combine pattern, 75 
“reduction abstain” flag, of combine 
interface, 28,81 

reduction messages, (via combine 
interface), 77 
register constants, 11 
casting, for C coding, 11 
register fields 
names, 10 

programming constants, 11 
register interface 

of asynchronous global interface, 91 
of broadcast interface, 64 
of combine interface, 72 
of Data Networks, 36 
of global interface, 89 
of supervisor asynch global interface, 93 
of synchronous global interface, 89 
register n amin g format, 

ni _interface _ purpose , 10 
register types, 9 
register 

address constants, 11 
doubleword operators, 136 
names, 10 
NI, 6 
status, 25 

relative addressing 
See also addressing 
translation to physical addressing, 109 
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Reset, NI, 14,117 

right Data Network interface (RDR), 2,35 
RISC microprocessor, of processing node, 

3 

router, 35 

See also Data Network 
router done complete 

Orange broadcast interrupt, 96 
Orange interrupt, 100 
“router done” flag. See “DR network done” 
flag 

router-done. See network done 
running NI programs. See programs 

S 

scan overflow, in addition scans, 78 
“scan overflow interrupt enable” flag, of 
combine interface, 79, 83 
“scan start” register, of combine interface, 
72,78 

sc anning 

addition scan overflow, 78 
combine interface operation, 71,77 
scanning with segments, 78 
segmented scanning, 78 
select address, for ch unk table addressing, 
109 

“self address”, of a processing node, 40 
“send combiner value” supervisor field, of 
combine interface, 83,84 
“send empty” flag 
of a network, 25,26 
of broadcast interface, 68 
of combine interface, 76 
send FIFO empty interrupt enable flag, 54 
“send FIFO enable” flag, of broadcast 
interface, 69,69 

“send length” supervisor field, of combine 
interface, 83,84 
“send ok” flag 

and discarded messages, 25 
of a network, 25,25 
of broadcast interface, 68 
of combine interface, 76 
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of Data Networks, 45,46 
“send pattern” supervisor field, of combine 
interface, 83,84 
send FIFO 

network registers for, 19 
of a network, 9,18,19 
“send space” field 
of a network, 25,26 
of broadcast interface, 68 
of combine interface, 76 
of Data Networks, 45,46 
“send start” supervisor field, of combine 
interface, 83,84 

“send state” field, of Data Network, 45,53 
“send stop” flag, of broadcast interface, 30, 
32 

“send” register 

of a network, 17,19 
of broadcast interface, 64,67,67 
of combine interface, 72,73 
using to clear the send FIFO, 83 
of Data Networks, 36,42 
send_f irst addresses 
calculating, 21 
constants, 21 

send_f±rst_long addresses, calculating, 
43 

“send-first long” register, of Data 
Networks, 36,43 
“send-first” register 
of a network, 17,19 
of broadcast interface, 64,67,67 
of combine interface, 72,73 
of Data Networks, 36,42 
sending 

a broadcast interface message, 66 
a combine interface message, 73 
a Data Network message, 42,43 
message modes, 40 
a global interface message, 92 
a network message, 18,19 
a network-done message, 79 
a reduction-scan message, 77 
a scan message, 77 


NI Vernon 2.2 (CM-5E), June 1994 
Copyright © 1994 Thinking Machines Corporation 






Concepts Index 


a synchronous global message, 90 
an asynch supervisor global message, 93 
an asynchronous global message, 92 
sending messages from nodes to PM, 121 
sending messages from PM to nodes, 120 
serial number (of NT), register, 116 
simulating arrival of a message, 24,142 
status pop register, of Data Networks, 36, 
47 

status register for long messages, of Data 
Networks, 36,45 

status register, alternate, of Data Networks, 
36,46 
status register 
fields and flags, 25 
of a network interface, 17,25 
of broadcast interface, 64,68 
of combine interface, 72,76 
of Data Networks, 36,45, 80 
register type, 9 
status registers 
accessor macro, 26 
reading, 26 

“stop send” flag, 107,116 
“stop” flag 

of a network, 30 
of broadcast interface, 69 
of combine interface, 83 
of Data Networks, 54 
supervisor area, of NI memory region, 7 
supervisor asynchronous global interface, 
of global interface, 88,89 
“supervisor asynchronous global” register, 
of supervisor asynch global 
interface, 89,93 

supervisor broadcast interface, 64 
See also broadcast interface 
supervisor message tags, 48 
supervisor operations, 7 

clearing combine send FIFO, 83 
clearing interface send FIFO, 31 
grabbing control of receive and status 
registers, 31 

reserving Data Network message tags, 48 
simulating arrival of a message, 24,142 
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triggering All Fall Down mode in DR, 55 
“synch global receive interrupt enable” 
flag, of synchronous global 
interface, 89,91,107 
“synchronous global completion” flag, of 
synchronous global interface, 89,90 
synchronous global receive interrupt, 91 
“synchronous global receive” flag, of 

synchronous global interface, 89,90 
synchronous interface, of global interface, 
88, 89 

T 

tag fields 

and interrupts, 48 
and message counting, 51 
of Data Network messages, 48 
tag value, of Data Network message, 42 
timer, NI. See Network Interface timer 
timer (NI), register, 113 
“timer enable” flag, 113 
timing, of NI operations, 135 
total length of message, 26 

U 

user area, of NI memory region, 7 
user broadcast interface, 64 
See also broadcast network 
user message tags, 48 
user programming model, 5 
“user receive interrupt mask” register, 38, 
49 

“user tag mask” register, 38,48 

V 

value, of a message, (single- or 
doubleword), 19 

virtual base address, of NI memory regions, 

8 

VU Programmer’s Handbook, xix 


263 





NI Programmer's Handbook 


W 

writing a message, 22 
writing a value to recieve register, to 
simulate arrival of message, 24 
writing registers, using doubleword 
operators, 136 

X 

XOR, combine operation, 75 

Y 

Yellow broadcast interrupt, 104 
Yellow interrupt, 96,100,169 
Bad memory (Bus Error) interrupt, 170 


bad memory (Bus Error) interrupt, 96, 
100 

Data Network message too long, 39,96, 
100,172 

on bad relative address, 41,100 
on broadcast/combine conflict, 73, 96, 
100,104,171 

on combine abstain flag error, 82, 96, 

100.170 

on illegal relative address, 96,172 
on negative DR message count, 52,96, 

100.171 

Yellow broadcast interrupt, 96,100,104, 
170 

“Yellow interrupt level” field, 103 
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